API tools faq
paste
Advertisement
yonex

rickdom126:/var/log/glusterfs/glusterfs01.log

yonex
Dec 16th, 2016
222
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 49.07 KB | None | 0 0
raw download clone embed print report
  1. [2016-12-15 11:21:23.231937] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49161: ping timer event already removed
  2. [2016-12-15 11:21:23.232051] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49159: ping timer event already removed
  3. [2016-12-15 11:21:23.232074] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  4. [2016-12-15 11:21:23.232086] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
  5. [2016-12-15 11:21:23.232097] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  6. [2016-12-15 11:21:23.232105] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret -1 [Transport endpoint is not connected]
  7. [2016-12-15 11:21:23.232174] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.11:49162: ping timer event already removed
  8. [2016-12-15 11:21:23.232271] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1f2)[0x3b3882c162] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x82)[0x3b384139f2] (--> /usr/lib64/libgfrpc.so.0[0x3b38413a42] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x483)[0x3b3840ff63] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x10f0a)[0x7f4bd1f89f0a] ))))) 0-: 192.168.0.12:49160: ping timer event already removed
  9. [2016-12-15 11:21:23.232621] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  10. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of / returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003405] and [2016-12-15 11:21:23.232515]
  11. [2016-12-15 11:21:23.232620] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  12. [2016-12-15 11:21:23.232646] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1 (8d99db43-bbd8-4c44-90e4-82aa1f1f5d82) [Transport endpoint is not connected]
  13. [2016-12-15 11:21:23.232654] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  14. [2016-12-15 11:21:23.232664] D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret -1 [Transport endpoint is not connected]
  15. [2016-12-15 11:21:23.233018] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  16. The message "D [MSGID: 0] [dht-common.c:995:dht_revalidate_cbk] 0-glusterfs01-dht: revalidate lookup of /db1 returned with op_ret 0 [Invalid argument]" repeated 18 times between [2016-12-15 11:19:48.003931] and [2016-12-15 11:21:23.232932]
  17. [2016-12-15 11:21:23.233017] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack on glusterfs01-client-1
  18. [2016-12-15 11:21:23.233345] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack with op_ret 0 [Invalid argument]
  19. [2016-12-15 11:21:23.233404] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  20. [2016-12-15 11:21:23.233414] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  21. [2016-12-15 11:21:23.233423] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  22. [2016-12-15 11:21:23.233429] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  23. [2016-12-15 11:21:23.234030] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack (gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2). Holes=1 overlaps=0
  24. [2016-12-15 11:21:23.234049] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack
  25. [2016-12-15 11:21:23.234560] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack, gfid = e2ab7d64-881e-4142-9c5c-bdc84484d1f2
  26. [2016-12-15 11:21:23.234903] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-glusterfs01-dht: Calling fresh lookup for /db1/stack/02 on glusterfs01-client-3
  27. [2016-12-15 11:21:23.235253] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-glusterfs01-dht: fresh_lookup returned for /db1/stack/02 with op_ret 0 [Invalid argument]
  28. [2016-12-15 11:21:23.235289] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  29. [2016-12-15 11:21:23.235308] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  30. [2016-12-15 11:21:23.235316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  31. [2016-12-15 11:21:23.235322] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  32. [2016-12-15 11:21:23.235664] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glusterfs01-dht: Found anomalies in /db1/stack/02 (gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e). Holes=1 overlaps=0
  33. [2016-12-15 11:21:23.235677] D [MSGID: 0] [dht-common.c:894:dht_lookup_dir_cbk] 0-glusterfs01-dht: fixing assignment on /db1/stack/02
  34. [2016-12-15 11:21:23.235690] W [MSGID: 109005] [dht-selfheal.c:2106:dht_selfheal_directory] 0-glusterfs01-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /db1/stack/02, gfid = 4dc916fe-c7d3-4541-b2cf-325b40d55d6e
  35. [2016-12-15 11:21:23.236880] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  36. [2016-12-15 11:21:23.236904] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  37. [2016-12-15 11:21:23.236926] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  38. [2016-12-15 11:21:23.236934] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  39. [2016-12-15 11:21:23.236976] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  40. [2016-12-15 11:21:23.236987] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  41. [2016-12-15 11:21:23.236996] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  42. [2016-12-15 11:21:23.237002] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  43. [2016-12-15 11:21:23.237363] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  44. [2016-12-15 11:21:23.237370] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  45. [2016-12-15 11:21:23.237409] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  46. [2016-12-15 11:21:23.237430] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  47. [2016-12-15 11:21:23.237462] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  48. [2016-12-15 11:21:23.237464] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  49. [2016-12-15 11:21:23.237474] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  50. [2016-12-15 11:21:23.237481] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  51. [2016-12-15 11:21:23.241455] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  52. [2016-12-15 11:21:23.241477] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  53. [2016-12-15 11:21:23.241487] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  54. [2016-12-15 11:21:23.241503] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  55. [2016-12-15 11:21:23.241511] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  56. [2016-12-15 11:21:23.241518] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  57. [2016-12-15 11:21:23.241635] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  58. [2016-12-15 11:21:23.241646] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  59. [2016-12-15 11:21:23.241653] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  60. [2016-12-15 11:21:23.241668] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  61. [2016-12-15 11:21:23.241708] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  62. [2016-12-15 11:21:23.241725] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  63. [2016-12-15 11:21:23.241747] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  64. [2016-12-15 11:21:23.241753] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  65. [2016-12-15 11:21:23.242127] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  66. [2016-12-15 11:21:23.242140] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  67. [2016-12-15 11:21:23.242165] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  68. [2016-12-15 11:21:23.242184] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  69. [2016-12-15 11:21:23.242230] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  70. [2016-12-15 11:21:23.242244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  71. [2016-12-15 11:21:23.242234] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  72. [2016-12-15 11:21:23.242256] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  73. [2016-12-15 11:21:23.242282] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  74. [2016-12-15 11:21:23.242297] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  75. [2016-12-15 11:21:23.242305] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  76. [2016-12-15 11:21:23.242311] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  77. [2016-12-15 11:21:23.242317] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  78. [2016-12-15 11:21:23.242322] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  79. [2016-12-15 11:21:23.242395] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  80. [2016-12-15 11:21:23.242405] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  81. [2016-12-15 11:21:23.242412] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  82. [2016-12-15 11:21:23.242417] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  83. [2016-12-15 11:21:23.242441] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  84. [2016-12-15 11:21:23.242448] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  85. [2016-12-15 11:21:23.242455] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  86. [2016-12-15 11:21:23.242471] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  87. [2016-12-15 11:21:23.242889] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  88. [2016-12-15 11:21:23.242901] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  89. [2016-12-15 11:21:23.242907] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  90. [2016-12-15 11:21:23.242917] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  91. [2016-12-15 11:21:23.242919] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  92. [2016-12-15 11:21:23.242926] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  93. [2016-12-15 11:21:23.242934] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  94. [2016-12-15 11:21:23.242955] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  95. [2016-12-15 11:21:23.242969] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  96. [2016-12-15 11:21:23.242976] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  97. [2016-12-15 11:21:23.242982] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  98. [2016-12-15 11:21:23.242989] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  99. [2016-12-15 11:21:23.242994] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  100. [2016-12-15 11:21:23.242999] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  101. [2016-12-15 11:21:23.243089] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  102. [2016-12-15 11:21:23.243107] W [MSGID: 114031] [client-rpc-fops.c:798:client3_3_statfs_cbk] 0-glusterfs01-client-2: remote operation failed [Transport endpoint is not connected]
  103. [2016-12-15 11:21:23.243115] D [MSGID: 0] [client-rpc-fops.c:801:client3_3_statfs_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d7188, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  104. [2016-12-15 11:21:23.243131] W [MSGID: 109075] [dht-diskusage.c:44:dht_du_info_cbk] 0-glusterfs01-dht: failed to get disk info from glusterfs01-client-2 [Transport endpoint is not connected]
  105. [2016-12-15 11:21:23.243161] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  106. [2016-12-15 11:21:23.243167] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  107. [2016-12-15 11:21:23.243181] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  108. [2016-12-15 11:21:23.243187] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  109. [2016-12-15 11:21:23.243336] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-0': avail_percent is: 98.00 and avail_space is: 23618243006464 and avail_inodes is: 99.00
  110. [2016-12-15 11:21:23.243354] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-3': avail_percent is: 98.00 and avail_space is: 23638485438464 and avail_inodes is: 99.00
  111. [2016-12-15 11:21:23.243368] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-4': avail_percent is: 98.00 and avail_space is: 23621921132544 and avail_inodes is: 99.00
  112. [2016-12-15 11:21:23.243382] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-glusterfs01-dht: subvolume 'glusterfs01-client-1': avail_percent is: 98.00 and avail_space is: 23643445485568 and avail_inodes is: 99.00
  113. [2016-12-15 11:21:23.243428] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  114. [2016-12-15 11:21:23.243446] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  115. [2016-12-15 11:21:23.243452] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  116. [2016-12-15 11:21:23.243458] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  117. [2016-12-15 11:21:23.243470] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755435: /db1/stack/02/13b5c202 => -1 (Input/output error)
  118. [2016-12-15 11:21:23.243559] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  119. [2016-12-15 11:21:23.243567] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  120. [2016-12-15 11:21:23.243575] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  121. [2016-12-15 11:21:23.243580] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  122. [2016-12-15 11:21:23.243617] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  123. [2016-12-15 11:21:23.243624] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  124. [2016-12-15 11:21:23.243631] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  125. [2016-12-15 11:21:23.243637] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  126. [2016-12-15 11:21:23.243968] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  127. [2016-12-15 11:21:23.243982] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  128. [2016-12-15 11:21:23.244010] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  129. [2016-12-15 11:21:23.244015] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  130. [2016-12-15 11:21:23.244021] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  131. [2016-12-15 11:21:23.244034] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  132. [2016-12-15 11:21:23.244054] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  133. [2016-12-15 11:21:23.244065] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  134. [2016-12-15 11:21:23.244079] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  135. [2016-12-15 11:21:23.244088] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  136. [2016-12-15 11:21:23.244094] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  137. [2016-12-15 11:21:23.244101] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  138. [2016-12-15 11:21:23.244107] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  139. [2016-12-15 11:21:23.244112] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  140. [2016-12-15 11:21:23.244203] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  141. [2016-12-15 11:21:23.244210] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  142. [2016-12-15 11:21:23.244217] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  143. [2016-12-15 11:21:23.244222] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  144. [2016-12-15 11:21:23.244244] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  145. [2016-12-15 11:21:23.244251] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  146. [2016-12-15 11:21:23.244258] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  147. [2016-12-15 11:21:23.244263] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  148. [2016-12-15 11:21:23.244648] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  149. [2016-12-15 11:21:23.244661] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  150. [2016-12-15 11:21:23.244662] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  151. [2016-12-15 11:21:23.244681] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  152. [2016-12-15 11:21:23.244682] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  153. [2016-12-15 11:21:23.244699] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  154. [2016-12-15 11:21:23.244712] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  155. [2016-12-15 11:21:23.244720] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  156. [2016-12-15 11:21:23.244733] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  157. [2016-12-15 11:21:23.244741] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  158. [2016-12-15 11:21:23.244747] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  159. [2016-12-15 11:21:23.244753] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  160. [2016-12-15 11:21:23.244759] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  161. [2016-12-15 11:21:23.244764] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  162. [2016-12-15 11:21:23.244827] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  163. [2016-12-15 11:21:23.244836] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  164. [2016-12-15 11:21:23.244843] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  165. [2016-12-15 11:21:23.244848] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  166. [2016-12-15 11:21:23.244872] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  167. [2016-12-15 11:21:23.244879] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  168. [2016-12-15 11:21:23.244887] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  169. [2016-12-15 11:21:23.244892] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  170. [2016-12-15 11:21:23.245245] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  171. [2016-12-15 11:21:23.245262] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  172. [2016-12-15 11:21:23.245288] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  173. [2016-12-15 11:21:23.245298] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  174. [2016-12-15 11:21:23.245300] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  175. [2016-12-15 11:21:23.245316] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  176. [2016-12-15 11:21:23.245316] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  177. [2016-12-15 11:21:23.245331] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  178. [2016-12-15 11:21:23.245345] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  179. [2016-12-15 11:21:23.245353] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  180. [2016-12-15 11:21:23.245359] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  181. [2016-12-15 11:21:23.245365] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  182. [2016-12-15 11:21:23.245371] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  183. [2016-12-15 11:21:23.245377] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  184. [2016-12-15 11:21:23.245424] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  185. [2016-12-15 11:21:23.245434] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  186. [2016-12-15 11:21:23.245441] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  187. [2016-12-15 11:21:23.245445] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  188. [2016-12-15 11:21:23.245470] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  189. [2016-12-15 11:21:23.245478] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  190. [2016-12-15 11:21:23.245486] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  191. [2016-12-15 11:21:23.245494] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  192. [2016-12-15 11:21:23.245849] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  193. [2016-12-15 11:21:23.245858] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  194. [2016-12-15 11:21:23.245910] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  195. [2016-12-15 11:21:23.245922] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  196. [2016-12-15 11:21:23.245925] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  197. [2016-12-15 11:21:23.245939] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  198. [2016-12-15 11:21:23.245951] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  199. [2016-12-15 11:21:23.245959] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  200. [2016-12-15 11:21:23.245973] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  201. [2016-12-15 11:21:23.245981] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  202. [2016-12-15 11:21:23.245987] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  203. [2016-12-15 11:21:23.245994] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  204. [2016-12-15 11:21:23.246000] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  205. [2016-12-15 11:21:23.246005] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
  206. [2016-12-15 11:21:23.246055] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  207. [2016-12-15 11:21:23.246064] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  208. [2016-12-15 11:21:23.246069] E [MSGID: 109011] [dht-common.c:7063:dht_create] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202
  209. [2016-12-15 11:21:23.246075] D [MSGID: 0] [dht-common.c:7146:dht_create] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: Input/output error [Input/output error]
  210. [2016-12-15 11:21:23.246084] D [MSGID: 0] [read-ahead.c:174:ra_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-read-ahead returned -1 error: Input/output error [Input/output error]
  211. [2016-12-15 11:21:23.246090] D [MSGID: 0] [io-cache.c:705:ioc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: Input/output error [Input/output error]
  212. [2016-12-15 11:21:23.246096] D [MSGID: 0] [md-cache.c:1486:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: Input/output error [Input/output error]
  213. [2016-12-15 11:21:23.246102] D [MSGID: 0] [io-stats.c:1871:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: Input/output error [Input/output error]
  214. [2016-12-15 11:21:23.246111] W [fuse-bridge.c:2003:fuse_create_cbk] 0-glusterfs-fuse: 755440: /db1/stack/02/13b5c202 => -1 (Input/output error)
  215. [2016-12-15 11:21:23.246178] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-glusterfs01-dht: no subvolume for hash (value) = 4080705371
  216. [2016-12-15 11:21:23.246186] D [MSGID: 0] [dht-helper.c:787:dht_subvol_get_hashed] 0-glusterfs01-dht: No hashed subvolume for path=/db1/stack/02/13b5c202
  217. [2016-12-15 11:21:23.246194] D [MSGID: 0] [dht-common.c:2658:dht_lookup] 0-glusterfs01-dht: no subvolume in layout for path=/db1/stack/02/13b5c202, checking on all the subvols to see if it is a directory
  218. [2016-12-15 11:21:23.246198] D [MSGID: 0] [dht-common.c:2672:dht_lookup] 0-glusterfs01-dht: Found null hashed subvol. Calling lookup on all nodes.
  219. [2016-12-15 11:21:23.246224] D [MSGID: 0] [client.c:256:client_submit_request] 0-glusterfs01-client-2: connection in disconnected state
  220. [2016-12-15 11:21:23.246231] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-glusterfs01-client-2: remote operation failed. Path: /db1/stack/02/13b5c202 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
  221. [2016-12-15 11:21:23.246239] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
  222. [2016-12-15 11:21:23.246244] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-2 returned error [Transport endpoint is not connected]
  223. [2016-12-15 11:21:23.246568] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-0 returned -1 error: No such file or directory [No such file or directory]
  224. [2016-12-15 11:21:23.246592] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-4 returned -1 error: No such file or directory [No such file or directory]
  225. [2016-12-15 11:21:23.246593] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-0 returned error [No such file or directory]
  226. [2016-12-15 11:21:23.246605] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-4 returned error [No such file or directory]
  227. [2016-12-15 11:21:23.246627] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-3 returned -1 error: No such file or directory [No such file or directory]
  228. [2016-12-15 11:21:23.246632] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-3 returned error [No such file or directory]
  229. [2016-12-15 11:21:23.246633] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-client-1 returned -1 error: No such file or directory [No such file or directory]
  230. [2016-12-15 11:21:23.246649] D [MSGID: 0] [dht-common.c:842:dht_lookup_dir_cbk] 0-glusterfs01-dht: lookup of /db1/stack/02/13b5c202 on glusterfs01-client-1 returned error [No such file or directory]
  231. [2016-12-15 11:21:23.246673] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x223e3) [0x7f4bd1f9b3e3] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x4512c) [0x7f4bd1d3612c] -->/usr/lib64/glusterfs/3.8.5/xlator/cluster/distribute.so(+0x99f0) [0x7f4bd1cfa9f0] ) 0-glusterfs01-dht: invalid argument: inode [Invalid argument]
  232. [2016-12-15 11:21:23.246685] D [MSGID: 0] [dht-common.c:912:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-dht returned -1 error: No such file or directory [No such file or directory]
  233. [2016-12-15 11:21:23.246695] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-io-cache returned -1 error: No such file or directory [No such file or directory]
  234. [2016-12-15 11:21:23.246704] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-quick-read returned -1 error: No such file or directory [No such file or directory]
  235. [2016-12-15 11:21:23.246713] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01-md-cache returned -1 error: No such file or directory [No such file or directory]
  236. [2016-12-15 11:21:23.246722] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f4bdc1d78c4, glusterfs01 returned -1 error: No such file or directory [No such file or directory]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!