Advertisement
yonex

rickdom126:/var/log/glusterfs/glusterfs01.log

Dec 16th, 2016
167
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2016-12-15 11:21:23.231937] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  2. [2016-12-15 11:21:23.232051] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  3. [2016-12-15 11:21:23.232074] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  4. [2016-12-15 11:21:23.232086] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
  5. [2016-12-15 11:21:23.232097] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  6. [2016-12-15 11:21:23.232105] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret -1 [Transport endpoint is not connected]
  7. [2016-12-15 11:21:23.232174] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  8. [2016-12-15 11:21:23.232271] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  9. [2016-12-15 11:21:23.232621] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  10. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003405] and [2016-12-15 11:21:23.232515]
  11. [2016-12-15 11:21:23.232620] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  12. [2016-12-15 11:21:23.232646] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1 (8d99db43-bbd8-4c44-90e4-82aa1f1f5d82) [Transport endpoint is not connected]
  13. [2016-12-15 11:21:23.232654] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  14. [2016-12-15 11:21:23.232664] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret -1 [Transport endpoint is not connected]
  15. [2016-12-15 11:21:23.233018] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  16. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003931] and [2016-12-15 11:21:23.232932]
  17. [2016-12-15 11:21:23.233017] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack on glusterfs01-client-1
  18. [2016-12-15 11:21:23.233345] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack with op_ret 0 [Invalid argument]
  19. [2016-12-15 11:21:23.233404] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  20. [2016-12-15 11:21:23.233414] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  21. [2016-12-15 11:21:23.233423] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  22. [2016-12-15 11:21:23.233429] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  23. [2016-12-15 11:21:23.234030] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack (gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2). Holes=1 overlaps=0
  24. [2016-12-15 11:21:23.234049] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack
  25. [2016-12-15 11:21:23.234560] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack, gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2
  26. [2016-12-15 11:21:23.234903] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack/02 on glusterfs01-client-3
  27. [2016-12-15 11:21:23.235253] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack/02 with op_ret 0 [Invalid argument]
  28. [2016-12-15 11:21:23.235289] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  29. [2016-12-15 11:21:23.235308] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  30. [2016-12-15 11:21:23.235316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  31. [2016-12-15 11:21:23.235322] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  32. [2016-12-15 11:21:23.235664] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack/02 (gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e). Holes=1 overlaps=0
  33. [2016-12-15 11:21:23.235677] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack/02
  34. [2016-12-15 11:21:23.235690] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack/02, gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e
  35. [2016-12-15 11:21:23.236880] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  36. [2016-12-15 11:21:23.236904] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  37. [2016-12-15 11:21:23.236926] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  38. [2016-12-15 11:21:23.236934] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  39. [2016-12-15 11:21:23.236976] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  40. [2016-12-15 11:21:23.236987] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  41. [2016-12-15 11:21:23.236996] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  42. [2016-12-15 11:21:23.237002] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  43. [2016-12-15 11:21:23.237363] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  44. [2016-12-15 11:21:23.237370] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  45. [2016-12-15 11:21:23.237409] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  46. [2016-12-15 11:21:23.237430] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  47. [2016-12-15 11:21:23.237462] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  48. [2016-12-15 11:21:23.237464] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  49. [2016-12-15 11:21:23.237474] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  50. [2016-12-15 11:21:23.237481] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  51. [2016-12-15 11:21:23.241455] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  52. [2016-12-15 11:21:23.241477] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  53. [2016-12-15 11:21:23.241487] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  54. [2016-12-15 11:21:23.241503] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  55. [2016-12-15 11:21:23.241511] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  56. [2016-12-15 11:21:23.241518] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  57. [2016-12-15 11:21:23.241635] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  58. [2016-12-15 11:21:23.241646] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  59. [2016-12-15 11:21:23.241653] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  60. [2016-12-15 11:21:23.241668] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  61. [2016-12-15 11:21:23.241708] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  62. [2016-12-15 11:21:23.241725] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  63. [2016-12-15 11:21:23.241747] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  64. [2016-12-15 11:21:23.241753] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  65. [2016-12-15 11:21:23.242127] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  66. [2016-12-15 11:21:23.242140] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  67. [2016-12-15 11:21:23.242165] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  68. [2016-12-15 11:21:23.242184] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  69. [2016-12-15 11:21:23.242230] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  70. [2016-12-15 11:21:23.242244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  71. [2016-12-15 11:21:23.242234] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  72. [2016-12-15 11:21:23.242256] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  73. [2016-12-15 11:21:23.242282] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  74. [2016-12-15 11:21:23.242297] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  75. [2016-12-15 11:21:23.242305] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  76. [2016-12-15 11:21:23.242311] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  77. [2016-12-15 11:21:23.242317] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  78. [2016-12-15 11:21:23.242322] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  79. [2016-12-15 11:21:23.242395] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  80. [2016-12-15 11:21:23.242405] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  81. [2016-12-15 11:21:23.242412] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  82. [2016-12-15 11:21:23.242417] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  83. [2016-12-15 11:21:23.242441] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  84. [2016-12-15 11:21:23.242448] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  85. [2016-12-15 11:21:23.242455] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  86. [2016-12-15 11:21:23.242471] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  87. [2016-12-15 11:21:23.242889] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  88. [2016-12-15 11:21:23.242901] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  89. [2016-12-15 11:21:23.242907] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  90. [2016-12-15 11:21:23.242917] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  91. [2016-12-15 11:21:23.242919] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  92. [2016-12-15 11:21:23.242926] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  93. [2016-12-15 11:21:23.242934] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  94. [2016-12-15 11:21:23.242955] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  95. [2016-12-15 11:21:23.242969] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  96. [2016-12-15 11:21:23.242976] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  97. [2016-12-15 11:21:23.242982] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  98. [2016-12-15 11:21:23.242989] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  99. [2016-12-15 11:21:23.242994] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  100. [2016-12-15 11:21:23.242999] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  101. [2016-12-15 11:21:23.243089] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  102. [2016-12-15 11:21:23.243107] W [MSGID: 114031] [client-rpc-fops.c:798:client3_3_statfs_cbk] 0-glusterfs01-client-2: remote operation failed [Transport endpoint is not connected]
  103. [2016-12-15 11:21:23.243115] D [MSGID: 0] [client-rpc-fops.c:801:client3_3_statfs_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d7188, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  104. [2016-12-15 11:21:23.243131] W [MSGID: 109075] [dht-diskusage.c:44:dht_du_info_cbk] 0-glusterfs01-dht: failed to get disk info from glusterfs01-client-2 [Transport endpoint is not connected]
  105. [2016-12-15 11:21:23.243161] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  106. [2016-12-15 11:21:23.243167] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  107. [2016-12-15 11:21:23.243181] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  108. [2016-12-15 11:21:23.243187] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  109. [2016-12-15 11:21:23.243336] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-0': avail_percent is: 98.00 and avail_space is: 23618243006464 and avail_inodes is: 99.00
  110. [2016-12-15 11:21:23.243354] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-3': avail_percent is: 98.00 and avail_space is: 23638485438464 and avail_inodes is: 99.00
  111. [2016-12-15 11:21:23.243368] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-4': avail_percent is: 98.00 and avail_space is: 23621921132544 and avail_inodes is: 99.00
  112. [2016-12-15 11:21:23.243382] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-1': avail_percent is: 98.00 and avail_space is: 23643445485568 and avail_inodes is: 99.00
  113. [2016-12-15 11:21:23.243428] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  114. [2016-12-15 11:21:23.243446] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  115. [2016-12-15 11:21:23.243452] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  116. [2016-12-15 11:21:23.243458] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  117. [2016-12-15 11:21:23.243470] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755435: /db1/stack/02/13b5c202 => -1 (Input/output error)
  118. [2016-12-15 11:21:23.243559] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  119. [2016-12-15 11:21:23.243567] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  120. [2016-12-15 11:21:23.243575] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  121. [2016-12-15 11:21:23.243580] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  122. [2016-12-15 11:21:23.243617] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  123. [2016-12-15 11:21:23.243624] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  124. [2016-12-15 11:21:23.243631] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  125. [2016-12-15 11:21:23.243637] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  126. [2016-12-15 11:21:23.243968] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  127. [2016-12-15 11:21:23.243982] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  128. [2016-12-15 11:21:23.244010] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  129. [2016-12-15 11:21:23.244015] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  130. [2016-12-15 11:21:23.244021] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  131. [2016-12-15 11:21:23.244034] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  132. [2016-12-15 11:21:23.244054] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  133. [2016-12-15 11:21:23.244065] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  134. [2016-12-15 11:21:23.244079] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  135. [2016-12-15 11:21:23.244088] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  136. [2016-12-15 11:21:23.244094] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  137. [2016-12-15 11:21:23.244101] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  138. [2016-12-15 11:21:23.244107] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  139. [2016-12-15 11:21:23.244112] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  140. [2016-12-15 11:21:23.244203] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  141. [2016-12-15 11:21:23.244210] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  142. [2016-12-15 11:21:23.244217] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  143. [2016-12-15 11:21:23.244222] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  144. [2016-12-15 11:21:23.244244] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  145. [2016-12-15 11:21:23.244251] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  146. [2016-12-15 11:21:23.244258] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  147. [2016-12-15 11:21:23.244263] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  148. [2016-12-15 11:21:23.244648] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  149. [2016-12-15 11:21:23.244661] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  150. [2016-12-15 11:21:23.244662] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  151. [2016-12-15 11:21:23.244681] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  152. [2016-12-15 11:21:23.244682] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  153. [2016-12-15 11:21:23.244699] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  154. [2016-12-15 11:21:23.244712] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  155. [2016-12-15 11:21:23.244720] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  156. [2016-12-15 11:21:23.244733] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  157. [2016-12-15 11:21:23.244741] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  158. [2016-12-15 11:21:23.244747] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  159. [2016-12-15 11:21:23.244753] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  160. [2016-12-15 11:21:23.244759] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  161. [2016-12-15 11:21:23.244764] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  162. [2016-12-15 11:21:23.244827] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  163. [2016-12-15 11:21:23.244836] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  164. [2016-12-15 11:21:23.244843] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  165. [2016-12-15 11:21:23.244848] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  166. [2016-12-15 11:21:23.244872] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  167. [2016-12-15 11:21:23.244879] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  168. [2016-12-15 11:21:23.244887] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  169. [2016-12-15 11:21:23.244892] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  170. [2016-12-15 11:21:23.245245] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  171. [2016-12-15 11:21:23.245262] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  172. [2016-12-15 11:21:23.245288] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  173. [2016-12-15 11:21:23.245298] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  174. [2016-12-15 11:21:23.245300] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  175. [2016-12-15 11:21:23.245316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  176. [2016-12-15 11:21:23.245316] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  177. [2016-12-15 11:21:23.245331] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  178. [2016-12-15 11:21:23.245345] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  179. [2016-12-15 11:21:23.245353] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  180. [2016-12-15 11:21:23.245359] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  181. [2016-12-15 11:21:23.245365] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  182. [2016-12-15 11:21:23.245371] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  183. [2016-12-15 11:21:23.245377] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  184. [2016-12-15 11:21:23.245424] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  185. [2016-12-15 11:21:23.245434] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  186. [2016-12-15 11:21:23.245441] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  187. [2016-12-15 11:21:23.245445] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  188. [2016-12-15 11:21:23.245470] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  189. [2016-12-15 11:21:23.245478] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  190. [2016-12-15 11:21:23.245486] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  191. [2016-12-15 11:21:23.245494] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  192. [2016-12-15 11:21:23.245849] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  193. [2016-12-15 11:21:23.245858] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  194. [2016-12-15 11:21:23.245910] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  195. [2016-12-15 11:21:23.245922] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  196. [2016-12-15 11:21:23.245925] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  197. [2016-12-15 11:21:23.245939] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  198. [2016-12-15 11:21:23.245951] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  199. [2016-12-15 11:21:23.245959] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  200. [2016-12-15 11:21:23.245973] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  201. [2016-12-15 11:21:23.245981] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  202. [2016-12-15 11:21:23.245987] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  203. [2016-12-15 11:21:23.245994] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  204. [2016-12-15 11:21:23.246000] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  205. [2016-12-15 11:21:23.246005] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  206. [2016-12-15 11:21:23.246055] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  207. [2016-12-15 11:21:23.246064] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  208. [2016-12-15 11:21:23.246069] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  209. [2016-12-15 11:21:23.246075] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  210. [2016-12-15 11:21:23.246084] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  211. [2016-12-15 11:21:23.246090] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  212. [2016-12-15 11:21:23.246096] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  213. [2016-12-15 11:21:23.246102] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  214. [2016-12-15 11:21:23.246111] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755440: /db1/stack/02/13b5c202 => -1 (Input/output error)
  215. [2016-12-15 11:21:23.246178] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  216. [2016-12-15 11:21:23.246186] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  217. [2016-12-15 11:21:23.246194] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  218. [2016-12-15 11:21:23.246198] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  219. [2016-12-15 11:21:23.246224] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  220. [2016-12-15 11:21:23.246231] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  221. [2016-12-15 11:21:23.246239] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  222. [2016-12-15 11:21:23.246244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  223. [2016-12-15 11:21:23.246568] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  224. [2016-12-15 11:21:23.246592] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  225. [2016-12-15 11:21:23.246593] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  226. [2016-12-15 11:21:23.246605] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  227. [2016-12-15 11:21:23.246627] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  228. [2016-12-15 11:21:23.246632] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  229. [2016-12-15 11:21:23.246633] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  230. [2016-12-15 11:21:23.246649] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  231. [2016-12-15 11:21:23.246673] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  232. [2016-12-15 11:21:23.246685] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  233. [2016-12-15 11:21:23.246695] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  234. [2016-12-15 11:21:23.246704] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  235. [2016-12-15 11:21:23.246713] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  236. [2016-12-15 11:21:23.246722] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
Advertisement
Advertisement
Advertisement
RAW Paste Data Copied
Advertisement