API tools faq
paste
Advertisement
yonex

glusterfs01.log around "disconnecting now"

yonex
Dec 19th, 2016
308
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 62.59 KB | None | 0 0
raw download clone embed print report
  1. [2016-12-15 11:20:17.888019] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
  2. [2016-12-15 11:20:17.888054] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
  3. [2016-12-15 11:20:17.888063] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
  4. [2016-12-15 11:20:17.888076] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
  5. [2016-12-15 11:20:17.888083] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-4: returning as transport is already disconnected OR there are no frames (0 || 0)
  6. [2016-12-15 11:20:35.854313] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  7. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  8. [2016-12-15 11:20:35.854405] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  9. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  10. [2016-12-15 11:20:35.854482] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  11. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  12. [2016-12-15 11:20:35.854556] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  13. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  14. [2016-12-15 11:20:35.854629] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  15. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  16. [2016-12-15 11:21:17.854249] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
  17. [2016-12-15 11:21:17.854284] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_DISCONNECT
  18. [2016-12-15 11:21:17.854295] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-glusterfs01-client-2: disconnected from glusterfs01-client-2. Client process will keep trying to connect to glusterd until brick's port is available
  19. [2016-12-15 11:21:17.854305] D [fuse-bridge.c:5264:notify] 0-fuse: got event 8 on graph 0
  20. [2016-12-15 11:21:17.854311] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=4 from gen=7 to slot->gen=8, fd=13, slot->fd=13
  21. [2016-12-15 11:21:17.893581] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
  22. [2016-12-15 11:21:17.893606] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
  23. [2016-12-15 11:21:17.893611] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
  24. [2016-12-15 11:21:17.893623] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-4: returning as transport is already disconnected OR there are no frames (0 || 0)
  25. [2016-12-15 11:21:23.231937] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  26. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  27. [2016-12-15 11:21:23.232051] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  28. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  29. [2016-12-15 11:21:23.232074] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  30. [2016-12-15 11:21:23.232086] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
  31. [2016-12-15 11:21:23.232097] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  32. [2016-12-15 11:21:23.232105] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret -1 [Transport endpoint is not connected]
  33. [2016-12-15 11:21:23.232174] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  34. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  35. [2016-12-15 11:21:23.232271] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  36. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  37. [2016-12-15 11:21:23.232621] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  38. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003405] and [2016-12-15 11:21:23.232515]
  39. [2016-12-15 11:21:23.232620] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  40. [2016-12-15 11:21:23.232646] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1 (8d99db43-bbd8-4c44-90e4-82aa1f1f5d82) [Transport endpoint is not connected]
  41. [2016-12-15 11:21:23.232654] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  42. [2016-12-15 11:21:23.232664] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret -1 [Transport endpoint is not connected]
  43. [2016-12-15 11:21:23.233018] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  44. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003931] and [2016-12-15 11:21:23.232932]
  45. [2016-12-15 11:21:23.233017] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack on glusterfs01-client-1
  46. [2016-12-15 11:21:23.233345] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack with op_ret 0 [Invalid argument]
  47. [2016-12-15 11:21:23.233404] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  48. [2016-12-15 11:21:23.233414] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  49. [2016-12-15 11:21:23.233423] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  50. [2016-12-15 11:21:23.233429] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  51. [2016-12-15 11:21:23.234030] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack (gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2). Holes=1 overlaps=0
  52. [2016-12-15 11:21:23.234049] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack
  53. [2016-12-15 11:21:23.234560] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack, gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2
  54. [2016-12-15 11:21:23.234903] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack/02 on glusterfs01-client-3
  55. [2016-12-15 11:21:23.235253] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack/02 with op_ret 0 [Invalid argument]
  56. [2016-12-15 11:21:23.235289] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  57. [2016-12-15 11:21:23.235308] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  58. [2016-12-15 11:21:23.235316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  59. [2016-12-15 11:21:23.235322] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  60. [2016-12-15 11:21:23.235664] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack/02 (gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e). Holes=1 overlaps=0
  61. [2016-12-15 11:21:23.235677] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack/02
  62. [2016-12-15 11:21:23.235690] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack/02, gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e
  63. [2016-12-15 11:21:23.236880] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  64. [2016-12-15 11:21:23.236904] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  65. [2016-12-15 11:21:23.236926] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  66. [2016-12-15 11:21:23.236934] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  67. [2016-12-15 11:21:23.236976] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  68. [2016-12-15 11:21:23.236987] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  69. [2016-12-15 11:21:23.236996] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  70. [2016-12-15 11:21:23.237002] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  71. [2016-12-15 11:21:23.237363] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  72. [2016-12-15 11:21:23.237370] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  73. [2016-12-15 11:21:23.237409] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  74. [2016-12-15 11:21:23.237430] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  75. [2016-12-15 11:21:23.237462] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  76. [2016-12-15 11:21:23.237464] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  77. [2016-12-15 11:21:23.237474] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  78. [2016-12-15 11:21:23.237481] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  79. [2016-12-15 11:21:23.241455] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid a
  80. rgument: inode [Invalid argument]
  81. [2016-12-15 11:21:23.241477] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  82. [2016-12-15 11:21:23.241487] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  83. [2016-12-15 11:21:23.241503] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  84. [2016-12-15 11:21:23.241511] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  85. [2016-12-15 11:21:23.241518] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  86. [2016-12-15 11:21:23.241635] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  87. [2016-12-15 11:21:23.241646] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  88. [2016-12-15 11:21:23.241653] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  89. [2016-12-15 11:21:23.241668] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  90. [2016-12-15 11:21:23.241708] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  91. [2016-12-15 11:21:23.241725] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  92. [2016-12-15 11:21:23.241747] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  93. [2016-12-15 11:21:23.241753] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  94. [2016-12-15 11:21:23.242127] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  95. [2016-12-15 11:21:23.242140] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  96. [2016-12-15 11:21:23.242165] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  97. [2016-12-15 11:21:23.242184] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  98. [2016-12-15 11:21:23.242230] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  99. [2016-12-15 11:21:23.242244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  100. [2016-12-15 11:21:23.242234] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  101. [2016-12-15 11:21:23.242256] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  102. [2016-12-15 11:21:23.242282] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  103. [2016-12-15 11:21:23.242297] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  104. [2016-12-15 11:21:23.242305] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  105. [2016-12-15 11:21:23.242311] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  106. [2016-12-15 11:21:23.242317] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  107. [2016-12-15 11:21:23.242322] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  108. [2016-12-15 11:21:23.242395] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  109. [2016-12-15 11:21:23.242405] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  110. [2016-12-15 11:21:23.242412] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  111. [2016-12-15 11:21:23.242417] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  112. [2016-12-15 11:21:23.242441] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  113. [2016-12-15 11:21:23.242448] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  114. [2016-12-15 11:21:23.242455] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  115. [2016-12-15 11:21:23.242471] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  116. [2016-12-15 11:21:23.242889] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  117. [2016-12-15 11:21:23.242901] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  118. [2016-12-15 11:21:23.242907] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  119. [2016-12-15 11:21:23.242917] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  120. [2016-12-15 11:21:23.242919] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  121. [2016-12-15 11:21:23.242926] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  122. [2016-12-15 11:21:23.242934] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  123. [2016-12-15 11:21:23.242955] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  124. [2016-12-15 11:21:23.242969] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  125. [2016-12-15 11:21:23.242976] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  126. [2016-12-15 11:21:23.242982] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  127. [2016-12-15 11:21:23.242989] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  128. [2016-12-15 11:21:23.242994] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  129. [2016-12-15 11:21:23.242999] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  130. [2016-12-15 11:21:23.243089] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  131. [2016-12-15 11:21:23.243107] W [MSGID: 114031] [client-rpc-fops.c:798:client3_3_statfs_cbk] 0-glusterfs01-client-2: remote operation failed [Transport endpoint is not connected]
  132. [2016-12-15 11:21:23.243115] D [MSGID: 0] [client-rpc-fops.c:801:client3_3_statfs_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d7188, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  133. [2016-12-15 11:21:23.243131] W [MSGID: 109075] [dht-diskusage.c:44:dht_du_info_cbk] 0-glusterfs01-dht: failed to get disk info from glusterfs01-client-2 [Transport endpoint is not connected]
  134. [2016-12-15 11:21:23.243161] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  135. [2016-12-15 11:21:23.243167] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  136. [2016-12-15 11:21:23.243181] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  137. [2016-12-15 11:21:23.243187] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  138. [2016-12-15 11:21:23.243336] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-0': avail_percent is: 98.00 and avail_space is: 23618243006464 and avail_inodes is: 99.00
  139. [2016-12-15 11:21:23.243354] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-3': avail_percent is: 98.00 and avail_space is: 23638485438464 and avail_inodes is: 99.00
  140. [2016-12-15 11:21:23.243368] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-4': avail_percent is: 98.00 and avail_space is: 23621921132544 and avail_inodes is: 99.00
  141. [2016-12-15 11:21:23.243382] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-1': avail_percent is: 98.00 and avail_space is: 23643445485568 and avail_inodes is: 99.00
  142. [2016-12-15 11:21:23.243428] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  143. [2016-12-15 11:21:23.243446] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  144. [2016-12-15 11:21:23.243452] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  145. [2016-12-15 11:21:23.243458] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  146. [2016-12-15 11:21:23.243470] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755435: /db1/stack/02/13b5c202 => -1 (Input/output error)
  147. [2016-12-15 11:21:23.243559] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  148. [2016-12-15 11:21:23.243567] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  149. [2016-12-15 11:21:23.243575] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  150. [2016-12-15 11:21:23.243580] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  151. [2016-12-15 11:21:23.243617] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  152. [2016-12-15 11:21:23.243624] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  153. [2016-12-15 11:21:23.243631] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  154. [2016-12-15 11:21:23.243637] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  155. [2016-12-15 11:21:23.243968] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  156. [2016-12-15 11:21:23.243982] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  157. [2016-12-15 11:21:23.244010] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  158. [2016-12-15 11:21:23.244015] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  159. [2016-12-15 11:21:23.244021] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  160. [2016-12-15 11:21:23.244034] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  161. [2016-12-15 11:21:23.244054] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  162. [2016-12-15 11:21:23.244065] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  163. [2016-12-15 11:21:23.244079] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  164. [2016-12-15 11:21:23.244088] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  165. [2016-12-15 11:21:23.244094] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  166. [2016-12-15 11:21:23.244101] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  167. [2016-12-15 11:21:23.244107] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  168. [2016-12-15 11:21:23.244112] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  169. [2016-12-15 11:21:23.244203] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  170. [2016-12-15 11:21:23.244210] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  171. [2016-12-15 11:21:23.244217] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  172. [2016-12-15 11:21:23.244222] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  173. [2016-12-15 11:21:23.244244] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  174. [2016-12-15 11:21:23.244251] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  175. [2016-12-15 11:21:23.244258] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  176. [2016-12-15 11:21:23.244263] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  177. [2016-12-15 11:21:23.244648] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  178. [2016-12-15 11:21:23.244661] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  179. [2016-12-15 11:21:23.244662] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  180. [2016-12-15 11:21:23.244681] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  181. [2016-12-15 11:21:23.244682] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  182. [2016-12-15 11:21:23.244699] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  183. [2016-12-15 11:21:23.244712] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  184. [2016-12-15 11:21:23.244720] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  185. [2016-12-15 11:21:23.244733] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  186. [2016-12-15 11:21:23.244741] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  187. [2016-12-15 11:21:23.244747] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  188. [2016-12-15 11:21:23.244753] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  189. [2016-12-15 11:21:23.244759] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  190. [2016-12-15 11:21:23.244764] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  191. [2016-12-15 11:21:23.244827] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  192. [2016-12-15 11:21:23.244836] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  193. [2016-12-15 11:21:23.244843] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  194. [2016-12-15 11:21:23.244848] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  195. [2016-12-15 11:21:23.244872] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  196. [2016-12-15 11:21:23.244879] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  197. [2016-12-15 11:21:23.244887] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  198. [2016-12-15 11:21:23.244892] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  199. [2016-12-15 11:21:23.245245] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  200. [2016-12-15 11:21:23.245262] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  201. [2016-12-15 11:21:23.245288] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  202. [2016-12-15 11:21:23.245298] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  203. [2016-12-15 11:21:23.245300] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  204. [2016-12-15 11:21:23.245316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  205. [2016-12-15 11:21:23.245316] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  206. [2016-12-15 11:21:23.245331] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  207. [2016-12-15 11:21:23.245345] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  208. [2016-12-15 11:21:23.245353] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  209. [2016-12-15 11:21:23.245359] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  210. [2016-12-15 11:21:23.245365] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  211. [2016-12-15 11:21:23.245371] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  212. [2016-12-15 11:21:23.245377] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  213. [2016-12-15 11:21:23.245424] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  214. [2016-12-15 11:21:23.245434] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  215. [2016-12-15 11:21:23.245441] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  216. [2016-12-15 11:21:23.245445] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  217. [2016-12-15 11:21:23.245470] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  218. [2016-12-15 11:21:23.245478] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  219. [2016-12-15 11:21:23.245486] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  220. [2016-12-15 11:21:23.245494] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  221. [2016-12-15 11:21:23.245849] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  222. [2016-12-15 11:21:23.245858] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  223. [2016-12-15 11:21:23.245910] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  224. [2016-12-15 11:21:23.245922] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  225. [2016-12-15 11:21:23.245925] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  226. [2016-12-15 11:21:23.245939] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  227. [2016-12-15 11:21:23.245951] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  228. [2016-12-15 11:21:23.245959] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  229. [2016-12-15 11:21:23.245973] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  230. [2016-12-15 11:21:23.245981] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  231. [2016-12-15 11:21:23.245987] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  232. [2016-12-15 11:21:23.245994] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  233. [2016-12-15 11:21:23.246000] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  234. [2016-12-15 11:21:23.246005] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  235. [2016-12-15 11:21:23.246055] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  236. [2016-12-15 11:21:23.246064] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  237. [2016-12-15 11:21:23.246069] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  238. [2016-12-15 11:21:23.246075] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  239. [2016-12-15 11:21:23.246084] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  240. [2016-12-15 11:21:23.246090] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  241. [2016-12-15 11:21:23.246096] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  242. [2016-12-15 11:21:23.246102] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  243. [2016-12-15 11:21:23.246111] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755440: /db1/stack/02/13b5c202 => -1 (Input/output error)
  244. [2016-12-15 11:21:23.246178] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  245. [2016-12-15 11:21:23.246186] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  246. [2016-12-15 11:21:23.246194] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  247. [2016-12-15 11:21:23.246198] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  248. [2016-12-15 11:21:23.246224] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  249. [2016-12-15 11:21:23.246231] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  250. [2016-12-15 11:21:23.246239] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  251. [2016-12-15 11:21:23.246244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  252. [2016-12-15 11:21:23.246568] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  253. [2016-12-15 11:21:23.246592] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  254. [2016-12-15 11:21:23.246593] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  255. [2016-12-15 11:21:23.246605] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  256. [2016-12-15 11:21:23.246627] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  257. [2016-12-15 11:21:23.246632] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  258. [2016-12-15 11:21:23.246633] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  259. [2016-12-15 11:21:23.246649] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  260. [2016-12-15 11:21:23.246673] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  261. [2016-12-15 11:21:23.246685] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  262. [2016-12-15 11:21:23.246695] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  263. [2016-12-15 11:21:23.246704] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  264. [2016-12-15 11:21:23.246713] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  265. [2016-12-15 11:21:23.246722] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  266. [2016-12-15 11:21:27.896070] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-192.168.0.13 (port-24007) for hostname: 192.168.0.13 and port: 24007
  267. [2016-12-15 11:21:27.896094] D [socket.c:2889:socket_fix_ssl_opts] 0-glusterfs01-client-2: disabling SSL for portmapper connection
  268. [2016-12-15 11:21:27.896328] D [MSGID: 0] [client.c:2235:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_CONNECT
  269. [2016-12-15 11:21:27.896461] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  270. [2016-12-15 11:21:27.896471] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  271. [2016-12-15 11:21:27.896740] D [MSGID: 0] [client-handshake.c:1480:server_has_portmap] 0-glusterfs01-client-2: detected portmapper on server
  272. [2016-12-15 11:21:27.896861] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  273. [2016-12-15 11:21:27.896870] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  274. [2016-12-15 11:21:27.896961] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-glusterfs01-client-2: changing port to 49161 (from 0)
  275. [2016-12-15 11:21:27.896975] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
  276. [2016-12-15 11:21:27.897033] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x70)[0x3b38410910] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1cf)[0x3b38410b5f] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x3b3840bc68] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  277. [2016-12-15 11:21:27.897046] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  278. [2016-12-15 11:21:27.897045] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_DISCONNECT
  279. [2016-12-15 11:21:27.897046] D [MSGID: 0] [client.c:2309:client_rpc_notify] 0-glusterfs01-client-2: disconnected (skipped notify)
  280. [2016-12-15 11:21:27.898589] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-192.168.0.13 (port-24007) for hostname: 192.168.0.13 and port: 24007
  281. [2016-12-15 11:21:27.898669] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=28 to slot->gen=29, fd=10, slot->fd=10
  282. [2016-12-15 11:21:27.898853] D [MSGID: 0] [client.c:2235:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_CONNECT
  283. [2016-12-15 11:21:27.898940] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  284. [2016-12-15 11:21:27.898949] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  285. [2016-12-15 11:21:27.899055] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-glusterfs01-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
  286. [2016-12-15 11:21:27.899136] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  287. [2016-12-15 11:21:27.899145] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  288. [2016-12-15 11:21:27.899455] D [MSGID: 0] [client-handshake.c:1194:client_setvolume_cbk] 0-glusterfs01-client-2: clnt-lk-version = 1, server-lk-version = 0
  289. [2016-12-15 11:21:27.899466] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-glusterfs01-client-2: Connected to glusterfs01-client-2, attached to remote volume '/var/disk1/brick'.
  290. [2016-12-15 11:21:27.899474] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-glusterfs01-client-2: Server and Client lk-version numbers are not same, reopening the fds
  291. [2016-12-15 11:21:27.899481] D [MSGID: 0] [client-handshake.c:1064:client_post_handshake] 0-glusterfs01-client-2: No fds to open - notifying all parents child up
  292. [2016-12-15 11:21:27.899486] D [MSGID: 0] [client-handshake.c:243:client_set_lk_version] 0-glusterfs01-client-2: Sending SET_LK_VERSION
  293. [2016-12-15 11:21:27.899550] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  294. [2016-12-15 11:21:27.899578] D [fuse-bridge.c:5264:notify] 0-fuse: got event 5 on graph 0
  295. [2016-12-15 11:21:27.899735] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-glusterfs01-client-2: Server lk version = 1
  296. [2016-12-15 11:21:27.899889] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-2': avail_percent is: 98.00 and avail_space is: 23644940414976 and avail_inodes is: 99.00
  297. [2016-12-15 11:21:35.897105] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
  298. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db2 returned with op_ret 0 [Invalid argument]" repeated 14 times between [2016-12-15 11:19:48.004455] and [2016-12-15 11:20:02.562139]
  299. [2016-12-15 11:21:35.897268] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
  300. [2016-12-15 11:21:40.275843] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  301. [2016-12-15 11:21:40.275868] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for / on glusterfs01-client-1
  302. [2016-12-15 11:21:40.276361] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for / with op_ret 0 [Invalid argument]
  303. [2016-12-15 11:21:40.276975] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /db1, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  304. [2016-12-15 11:21:40.276989] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1 on glusterfs01-client-1
  305. [2016-12-15 11:21:40.277293] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1 with op_ret 0 [Invalid argument]
  306. [2016-12-15 11:21:40.277719] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /db2, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  307. [2016-12-15 11:21:40.277734] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db2 on glusterfs01-client-1
  308. [2016-12-15 11:21:40.278084] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db2 with op_ret 0 [Invalid argument]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!