Advertisement
yonex

glusterfs01.log around "disconnecting now"

Dec 19th, 2016
210
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2016-12-15 11:20:17.888019] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
  2. [2016-12-15 11:20:17.888054] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
  3. [2016-12-15 11:20:17.888063] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
  4. [2016-12-15 11:20:17.888076] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
  5. [2016-12-15 11:20:17.888083] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-4: returning as transport is already disconnected OR there are no frames (0 || 0)
  6. [2016-12-15 11:20:35.854313] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  7. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  8. [2016-12-15 11:20:35.854405] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  9. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  10. [2016-12-15 11:20:35.854482] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  11. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  12. [2016-12-15 11:20:35.854556] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  13. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  14. [2016-12-15 11:20:35.854629] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  15. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  16. [2016-12-15 11:21:17.854249] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
  17. [2016-12-15 11:21:17.854284] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_DISCONNECT
  18. [2016-12-15 11:21:17.854295] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-glusterfs01-client-2: disconnected from glusterfs01-client-2. Client process will keep trying to connect to glusterd until brick's port is available
  19. [2016-12-15 11:21:17.854305] D [fuse-bridge.c:5264:notify] 0-fuse: got event 8 on graph 0
  20. [2016-12-15 11:21:17.854311] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=4 from gen=7 to slot->gen=8, fd=13, slot->fd=13
  21. [2016-12-15 11:21:17.893581] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
  22. [2016-12-15 11:21:17.893606] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
  23. [2016-12-15 11:21:17.893611] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
  24. [2016-12-15 11:21:17.893623] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-4: returning as transport is already disconnected OR there are no frames (0 || 0)
  25. [2016-12-15 11:21:23.231937] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  26. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  27. [2016-12-15 11:21:23.232051] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  28. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  29. [2016-12-15 11:21:23.232074] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  30. [2016-12-15 11:21:23.232086] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
  31. [2016-12-15 11:21:23.232097] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  32. [2016-12-15 11:21:23.232105] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret -1 [Transport endpoint is not connected]
  33. [2016-12-15 11:21:23.232174] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  34. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  35. [2016-12-15 11:21:23.232271] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/
  36. lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  37. [2016-12-15 11:21:23.232621] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  38. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003405] and [2016-12-15 11:21:23.232515]
  39. [2016-12-15 11:21:23.232620] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  40. [2016-12-15 11:21:23.232646] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1 (8d99db43-bbd8-4c44-90e4-82aa1f1f5d82) [Transport endpoint is not connected]
  41. [2016-12-15 11:21:23.232654] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  42. [2016-12-15 11:21:23.232664] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret -1 [Transport endpoint is not connected]
  43. [2016-12-15 11:21:23.233018] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  44. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003931] and [2016-12-15 11:21:23.232932]
  45. [2016-12-15 11:21:23.233017] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack on glusterfs01-client-1
  46. [2016-12-15 11:21:23.233345] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack with op_ret 0 [Invalid argument]
  47. [2016-12-15 11:21:23.233404] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  48. [2016-12-15 11:21:23.233414] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  49. [2016-12-15 11:21:23.233423] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  50. [2016-12-15 11:21:23.233429] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  51. [2016-12-15 11:21:23.234030] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack (gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2). Holes=1 overlaps=0
  52. [2016-12-15 11:21:23.234049] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack
  53. [2016-12-15 11:21:23.234560] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack, gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2
  54. [2016-12-15 11:21:23.234903] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack/02 on glusterfs01-client-3
  55. [2016-12-15 11:21:23.235253] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack/02 with op_ret 0 [Invalid argument]
  56. [2016-12-15 11:21:23.235289] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  57. [2016-12-15 11:21:23.235308] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  58. [2016-12-15 11:21:23.235316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  59. [2016-12-15 11:21:23.235322] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  60. [2016-12-15 11:21:23.235664] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack/02 (gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e). Holes=1 overlaps=0
  61. [2016-12-15 11:21:23.235677] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack/02
  62. [2016-12-15 11:21:23.235690] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack/02, gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e
  63. [2016-12-15 11:21:23.236880] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  64. [2016-12-15 11:21:23.236904] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  65. [2016-12-15 11:21:23.236926] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  66. [2016-12-15 11:21:23.236934] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  67. [2016-12-15 11:21:23.236976] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  68. [2016-12-15 11:21:23.236987] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  69. [2016-12-15 11:21:23.236996] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  70. [2016-12-15 11:21:23.237002] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  71. [2016-12-15 11:21:23.237363] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  72. [2016-12-15 11:21:23.237370] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  73. [2016-12-15 11:21:23.237409] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  74. [2016-12-15 11:21:23.237430] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  75. [2016-12-15 11:21:23.237462] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  76. [2016-12-15 11:21:23.237464] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  77. [2016-12-15 11:21:23.237474] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  78. [2016-12-15 11:21:23.237481] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  79. [2016-12-15 11:21:23.241455] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid a
  80. rgument: inode [Invalid argument]
  81. [2016-12-15 11:21:23.241477] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  82. [2016-12-15 11:21:23.241487] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  83. [2016-12-15 11:21:23.241503] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  84. [2016-12-15 11:21:23.241511] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  85. [2016-12-15 11:21:23.241518] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  86. [2016-12-15 11:21:23.241635] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  87. [2016-12-15 11:21:23.241646] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  88. [2016-12-15 11:21:23.241653] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  89. [2016-12-15 11:21:23.241668] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  90. [2016-12-15 11:21:23.241708] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  91. [2016-12-15 11:21:23.241725] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  92. [2016-12-15 11:21:23.241747] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  93. [2016-12-15 11:21:23.241753] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  94. [2016-12-15 11:21:23.242127] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  95. [2016-12-15 11:21:23.242140] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  96. [2016-12-15 11:21:23.242165] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  97. [2016-12-15 11:21:23.242184] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  98. [2016-12-15 11:21:23.242230] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  99. [2016-12-15 11:21:23.242244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  100. [2016-12-15 11:21:23.242234] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  101. [2016-12-15 11:21:23.242256] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  102. [2016-12-15 11:21:23.242282] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  103. [2016-12-15 11:21:23.242297] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  104. [2016-12-15 11:21:23.242305] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  105. [2016-12-15 11:21:23.242311] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  106. [2016-12-15 11:21:23.242317] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  107. [2016-12-15 11:21:23.242322] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  108. [2016-12-15 11:21:23.242395] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  109. [2016-12-15 11:21:23.242405] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  110. [2016-12-15 11:21:23.242412] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  111. [2016-12-15 11:21:23.242417] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  112. [2016-12-15 11:21:23.242441] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  113. [2016-12-15 11:21:23.242448] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  114. [2016-12-15 11:21:23.242455] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  115. [2016-12-15 11:21:23.242471] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  116. [2016-12-15 11:21:23.242889] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  117. [2016-12-15 11:21:23.242901] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  118. [2016-12-15 11:21:23.242907] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  119. [2016-12-15 11:21:23.242917] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  120. [2016-12-15 11:21:23.242919] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  121. [2016-12-15 11:21:23.242926] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  122. [2016-12-15 11:21:23.242934] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  123. [2016-12-15 11:21:23.242955] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  124. [2016-12-15 11:21:23.242969] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  125. [2016-12-15 11:21:23.242976] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  126. [2016-12-15 11:21:23.242982] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  127. [2016-12-15 11:21:23.242989] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  128. [2016-12-15 11:21:23.242994] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  129. [2016-12-15 11:21:23.242999] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  130. [2016-12-15 11:21:23.243089] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  131. [2016-12-15 11:21:23.243107] W [MSGID: 114031] [client-rpc-fops.c:798:client3_3_statfs_cbk] 0-glusterfs01-client-2: remote operation failed [Transport endpoint is not connected]
  132. [2016-12-15 11:21:23.243115] D [MSGID: 0] [client-rpc-fops.c:801:client3_3_statfs_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d7188, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  133. [2016-12-15 11:21:23.243131] W [MSGID: 109075] [dht-diskusage.c:44:dht_du_info_cbk] 0-glusterfs01-dht: failed to get disk info from glusterfs01-client-2 [Transport endpoint is not connected]
  134. [2016-12-15 11:21:23.243161] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  135. [2016-12-15 11:21:23.243167] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  136. [2016-12-15 11:21:23.243181] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  137. [2016-12-15 11:21:23.243187] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  138. [2016-12-15 11:21:23.243336] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-0': avail_percent is: 98.00 and avail_space is: 23618243006464 and avail_inodes is: 99.00
  139. [2016-12-15 11:21:23.243354] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-3': avail_percent is: 98.00 and avail_space is: 23638485438464 and avail_inodes is: 99.00
  140. [2016-12-15 11:21:23.243368] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-4': avail_percent is: 98.00 and avail_space is: 23621921132544 and avail_inodes is: 99.00
  141. [2016-12-15 11:21:23.243382] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-1': avail_percent is: 98.00 and avail_space is: 23643445485568 and avail_inodes is: 99.00
  142. [2016-12-15 11:21:23.243428] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  143. [2016-12-15 11:21:23.243446] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  144. [2016-12-15 11:21:23.243452] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  145. [2016-12-15 11:21:23.243458] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  146. [2016-12-15 11:21:23.243470] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755435: /db1/stack/02/13b5c202 => -1 (Input/output error)
  147. [2016-12-15 11:21:23.243559] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  148. [2016-12-15 11:21:23.243567] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  149. [2016-12-15 11:21:23.243575] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  150. [2016-12-15 11:21:23.243580] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  151. [2016-12-15 11:21:23.243617] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  152. [2016-12-15 11:21:23.243624] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  153. [2016-12-15 11:21:23.243631] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  154. [2016-12-15 11:21:23.243637] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  155. [2016-12-15 11:21:23.243968] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  156. [2016-12-15 11:21:23.243982] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  157. [2016-12-15 11:21:23.244010] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  158. [2016-12-15 11:21:23.244015] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  159. [2016-12-15 11:21:23.244021] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  160. [2016-12-15 11:21:23.244034] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  161. [2016-12-15 11:21:23.244054] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  162. [2016-12-15 11:21:23.244065] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  163. [2016-12-15 11:21:23.244079] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  164. [2016-12-15 11:21:23.244088] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  165. [2016-12-15 11:21:23.244094] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  166. [2016-12-15 11:21:23.244101] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  167. [2016-12-15 11:21:23.244107] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  168. [2016-12-15 11:21:23.244112] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  169. [2016-12-15 11:21:23.244203] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  170. [2016-12-15 11:21:23.244210] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  171. [2016-12-15 11:21:23.244217] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  172. [2016-12-15 11:21:23.244222] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  173. [2016-12-15 11:21:23.244244] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  174. [2016-12-15 11:21:23.244251] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  175. [2016-12-15 11:21:23.244258] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  176. [2016-12-15 11:21:23.244263] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  177. [2016-12-15 11:21:23.244648] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  178. [2016-12-15 11:21:23.244661] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  179. [2016-12-15 11:21:23.244662] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  180. [2016-12-15 11:21:23.244681] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  181. [2016-12-15 11:21:23.244682] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  182. [2016-12-15 11:21:23.244699] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  183. [2016-12-15 11:21:23.244712] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  184. [2016-12-15 11:21:23.244720] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  185. [2016-12-15 11:21:23.244733] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  186. [2016-12-15 11:21:23.244741] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  187. [2016-12-15 11:21:23.244747] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  188. [2016-12-15 11:21:23.244753] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  189. [2016-12-15 11:21:23.244759] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  190. [2016-12-15 11:21:23.244764] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  191. [2016-12-15 11:21:23.244827] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  192. [2016-12-15 11:21:23.244836] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  193. [2016-12-15 11:21:23.244843] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  194. [2016-12-15 11:21:23.244848] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  195. [2016-12-15 11:21:23.244872] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  196. [2016-12-15 11:21:23.244879] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  197. [2016-12-15 11:21:23.244887] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  198. [2016-12-15 11:21:23.244892] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  199. [2016-12-15 11:21:23.245245] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  200. [2016-12-15 11:21:23.245262] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  201. [2016-12-15 11:21:23.245288] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  202. [2016-12-15 11:21:23.245298] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  203. [2016-12-15 11:21:23.245300] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  204. [2016-12-15 11:21:23.245316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  205. [2016-12-15 11:21:23.245316] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  206. [2016-12-15 11:21:23.245331] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  207. [2016-12-15 11:21:23.245345] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  208. [2016-12-15 11:21:23.245353] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  209. [2016-12-15 11:21:23.245359] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  210. [2016-12-15 11:21:23.245365] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  211. [2016-12-15 11:21:23.245371] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  212. [2016-12-15 11:21:23.245377] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  213. [2016-12-15 11:21:23.245424] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  214. [2016-12-15 11:21:23.245434] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  215. [2016-12-15 11:21:23.245441] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  216. [2016-12-15 11:21:23.245445] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  217. [2016-12-15 11:21:23.245470] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  218. [2016-12-15 11:21:23.245478] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  219. [2016-12-15 11:21:23.245486] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  220. [2016-12-15 11:21:23.245494] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  221. [2016-12-15 11:21:23.245849] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  222. [2016-12-15 11:21:23.245858] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  223. [2016-12-15 11:21:23.245910] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  224. [2016-12-15 11:21:23.245922] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  225. [2016-12-15 11:21:23.245925] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  226. [2016-12-15 11:21:23.245939] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  227. [2016-12-15 11:21:23.245951] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  228. [2016-12-15 11:21:23.245959] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  229. [2016-12-15 11:21:23.245973] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  230. [2016-12-15 11:21:23.245981] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  231. [2016-12-15 11:21:23.245987] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  232. [2016-12-15 11:21:23.245994] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  233. [2016-12-15 11:21:23.246000] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  234. [2016-12-15 11:21:23.246005] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  235. [2016-12-15 11:21:23.246055] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  236. [2016-12-15 11:21:23.246064] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  237. [2016-12-15 11:21:23.246069] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  238. [2016-12-15 11:21:23.246075] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  239. [2016-12-15 11:21:23.246084] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  240. [2016-12-15 11:21:23.246090] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  241. [2016-12-15 11:21:23.246096] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  242. [2016-12-15 11:21:23.246102] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  243. [2016-12-15 11:21:23.246111] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755440: /db1/stack/02/13b5c202 => -1 (Input/output error)
  244. [2016-12-15 11:21:23.246178] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  245. [2016-12-15 11:21:23.246186] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  246. [2016-12-15 11:21:23.246194] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  247. [2016-12-15 11:21:23.246198] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  248. [2016-12-15 11:21:23.246224] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  249. [2016-12-15 11:21:23.246231] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  250. [2016-12-15 11:21:23.246239] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  251. [2016-12-15 11:21:23.246244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  252. [2016-12-15 11:21:23.246568] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  253. [2016-12-15 11:21:23.246592] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  254. [2016-12-15 11:21:23.246593] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  255. [2016-12-15 11:21:23.246605] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  256. [2016-12-15 11:21:23.246627] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  257. [2016-12-15 11:21:23.246632] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  258. [2016-12-15 11:21:23.246633] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  259. [2016-12-15 11:21:23.246649] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  260. [2016-12-15 11:21:23.246673] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  261. [2016-12-15 11:21:23.246685] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  262. [2016-12-15 11:21:23.246695] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  263. [2016-12-15 11:21:23.246704] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  264. [2016-12-15 11:21:23.246713] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  265. [2016-12-15 11:21:23.246722] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  266. [2016-12-15 11:21:27.896070] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-192.168.0.13 (port-24007) for hostname: 192.168.0.13 and port: 24007
  267. [2016-12-15 11:21:27.896094] D [socket.c:2889:socket_fix_ssl_opts] 0-glusterfs01-client-2: disabling SSL for portmapper connection
  268. [2016-12-15 11:21:27.896328] D [MSGID: 0] [client.c:2235:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_CONNECT
  269. [2016-12-15 11:21:27.896461] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  270. [2016-12-15 11:21:27.896471] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  271. [2016-12-15 11:21:27.896740] D [MSGID: 0] [client-handshake.c:1480:server_has_portmap] 0-glusterfs01-client-2: detected portmapper on server
  272. [2016-12-15 11:21:27.896861] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  273. [2016-12-15 11:21:27.896870] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  274. [2016-12-15 11:21:27.896961] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-glusterfs01-client-2: changing port to 49161 (from 0)
  275. [2016-12-15 11:21:27.896975] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
  276. [2016-12-15 11:21:27.897033] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x70)[0x3b38410910] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1cf)[0x3b38410b5f] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x3b3840bc68] ))))) 0-: 192.168.0.13:24007: ping timer event already removed
  277. [2016-12-15 11:21:27.897046] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  278. [2016-12-15 11:21:27.897045] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_DISCONNECT
  279. [2016-12-15 11:21:27.897046] D [MSGID: 0] [client.c:2309:client_rpc_notify] 0-glusterfs01-client-2: disconnected (skipped notify)
  280. [2016-12-15 11:21:27.898589] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-192.168.0.13 (port-24007) for hostname: 192.168.0.13 and port: 24007
  281. [2016-12-15 11:21:27.898669] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=28 to slot->gen=29, fd=10, slot->fd=10
  282. [2016-12-15 11:21:27.898853] D [MSGID: 0] [client.c:2235:client_rpc_notify] 0-glusterfs01-client-2: got RPC_CLNT_CONNECT
  283. [2016-12-15 11:21:27.898940] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  284. [2016-12-15 11:21:27.898949] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  285. [2016-12-15 11:21:27.899055] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-glusterfs01-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
  286. [2016-12-15 11:21:27.899136] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  287. [2016-12-15 11:21:27.899145] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-glusterfs01-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
  288. [2016-12-15 11:21:27.899455] D [MSGID: 0] [client-handshake.c:1194:client_setvolume_cbk] 0-glusterfs01-client-2: clnt-lk-version = 1, server-lk-version = 0
  289. [2016-12-15 11:21:27.899466] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-glusterfs01-client-2: Connected to glusterfs01-client-2, attached to remote volume '/var/disk1/brick'.
  290. [2016-12-15 11:21:27.899474] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-glusterfs01-client-2: Server and Client lk-version numbers are not same, reopening the fds
  291. [2016-12-15 11:21:27.899481] D [MSGID: 0] [client-handshake.c:1064:client_post_handshake] 0-glusterfs01-client-2: No fds to open - notifying all parents child up
  292. [2016-12-15 11:21:27.899486] D [MSGID: 0] [client-handshake.c:243:client_set_lk_version] 0-glusterfs01-client-2: Sending SET_LK_VERSION
  293. [2016-12-15 11:21:27.899550] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.13:49161: ping timer event already removed
  294. [2016-12-15 11:21:27.899578] D [fuse-bridge.c:5264:notify] 0-fuse: got event 5 on graph 0
  295. [2016-12-15 11:21:27.899735] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-glusterfs01-client-2: Server lk version = 1
  296. [2016-12-15 11:21:27.899889] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-2': avail_percent is: 98.00 and avail_space is: 23644940414976 and avail_inodes is: 99.00
  297. [2016-12-15 11:21:35.897105] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
  298. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db2 returned with op_ret 0 [Invalid argument]" repeated 14 times between [2016-12-15 11:19:48.004455] and [2016-12-15 11:20:02.562139]
  299. [2016-12-15 11:21:35.897268] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
  300. [2016-12-15 11:21:40.275843] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  301. [2016-12-15 11:21:40.275868] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for / on glusterfs01-client-1
  302. [2016-12-15 11:21:40.276361] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for / with op_ret 0 [Invalid argument]
  303. [2016-12-15 11:21:40.276975] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /db1, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  304. [2016-12-15 11:21:40.276989] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1 on glusterfs01-client-1
  305. [2016-12-15 11:21:40.277293] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1 with op_ret 0 [Invalid argument]
  306. [2016-12-15 11:21:40.277719] D [MSGID: 0] [dht-common.c:2550:dht_lookup] 0-glusterfs01-dht: Called revalidate lookup for /db2, but layout->gen (29) is less than conf->gen (30), calling fresh_lookup
  307. [2016-12-15 11:21:40.277734] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db2 on glusterfs01-client-1
  308. [2016-12-15 11:21:40.278084] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db2 with op_ret 0 [Invalid argument]
Advertisement
Advertisement
Advertisement
RAW Paste Data Copied
Advertisement