yonex

192.168.0.11:/var/log/glusterfs/bricks/var-disk2-brick.log

Dec 16th, 2016
121
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2016-12-15 11:21:23.230994] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  2. [2016-12-15 11:21:23.231154] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  3. [2016-12-15 11:21:23.231365] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  4. [2016-12-15 11:21:23.231486] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  5. [2016-12-15 11:21:23.231510] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  6. [2016-12-15 11:21:23.231620] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  7. [2016-12-15 11:21:23.231979] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  8. [2016-12-15 11:21:23.232056] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  9. The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop" repeated 26 times between [2016-12-15 11:21:21.211683] and [2016-12-15 11:21:23.232008]
  10. [2016-12-15 11:21:23.232055] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  11. [2016-12-15 11:21:23.232091] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  12. [2016-12-15 11:21:23.232103] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  13. [2016-12-15 11:21:23.232112] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  14. [2016-12-15 11:21:23.232122] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  15. [2016-12-15 11:21:23.232133] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  16. [2016-12-15 11:21:23.232152] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  17. [2016-12-15 11:21:23.232170] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  18. [2016-12-15 11:21:23.232195] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  19. [2016-12-15 11:21:23.232213] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  20. [2016-12-15 11:21:23.232232] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 3227: LOOKUP /db2/trash/13b5c7f8.mp4 (cf85dcba-1349-45e0-ac45-b39433e543f0/13b5c7f8.mp4) ==> (No such file or directory) [No such file or directory]
  21. [2016-12-15 11:21:23.232250] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  22. [2016-12-15 11:21:23.232278] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  23. [2016-12-15 11:21:23.232303] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  24. [2016-12-15 11:21:23.232422] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  25. [2016-12-15 11:21:23.232457] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  26. [2016-12-15 11:21:23.232521] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  27. [2016-12-15 11:21:23.232485] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  28. [2016-12-15 11:21:23.232521] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  29. [2016-12-15 11:21:23.232550] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  30. [2016-12-15 11:21:23.232560] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  31. [2016-12-15 11:21:23.232569] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  32. [2016-12-15 11:21:23.232578] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  33. [2016-12-15 11:21:23.232587] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  34. [2016-12-15 11:21:23.232596] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  35. [2016-12-15 11:21:23.232606] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  36. [2016-12-15 11:21:23.232615] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  37. [2016-12-15 11:21:23.232624] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  38. [2016-12-15 11:21:23.232635] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 3228: LOOKUP /db2/trash/13b5c7f8.mp4 (cf85dcba-1349-45e0-ac45-b39433e543f0/13b5c7f8.mp4) ==> (No such file or directory) [No such file or directory]
  39. [2016-12-15 11:21:23.232684] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  40. [2016-12-15 11:21:23.233196] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x182fe) [0x7fef1376d2fe] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  41. [2016-12-15 11:21:23.233267] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2db65) [0x7fef13782b65] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 3
  42. [2016-12-15 11:21:23.233289] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  43. [2016-12-15 11:21:23.233223] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: STATFS scheduled as fast fop
  44. [2016-12-15 11:21:23.233289] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: CREATE scheduled as normal fop
  45. [2016-12-15 11:21:23.233386] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c272) [0x7fef13781272] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  46. [2016-12-15 11:21:23.233761] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  47. [2016-12-15 11:21:23.233786] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  48. [2016-12-15 11:21:23.233846] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x254a8) [0x7fef1377a4a8] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  49. [2016-12-15 11:21:23.233930] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  50. [2016-12-15 11:21:23.234144] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  51. [2016-12-15 11:21:23.234312] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  52. [2016-12-15 11:21:23.235818] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  53. [2016-12-15 11:21:23.235885] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
  54. The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop" repeated 2 times between [2016-12-15 11:21:23.233786] and [2016-12-15 11:21:23.235849]
  55. [2016-12-15 11:21:23.235885] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  56. [2016-12-15 11:21:23.235914] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  57. [2016-12-15 11:21:23.235930] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  58. [2016-12-15 11:21:23.235940] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  59. [2016-12-15 11:21:23.235950] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  60. [2016-12-15 11:21:23.235959] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  61. [2016-12-15 11:21:23.235968] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  62. [2016-12-15 11:21:23.235977] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  63. [2016-12-15 11:21:23.235986] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  64. [2016-12-15 11:21:23.235995] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  65. [2016-12-15 11:21:23.236005] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31416: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  66. [2016-12-15 11:21:23.236052] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  67. [2016-12-15 11:21:23.236382] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x182fe) [0x7fef1376d2fe] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom53.example.com-9040-2016/12/14-13:29:01:172149-glusterfs01-client-3-0-8: ref-count 2
  68. [2016-12-15 11:21:23.236401] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: STATFS scheduled as fast fop
  69. [2016-12-15 11:21:23.236502] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c272) [0x7fef13781272] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom53.example.com-9040-2016/12/14-13:29:01:172149-glusterfs01-client-3-0-8: ref-count 1
  70. [2016-12-15 11:21:23.240585] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  71. [2016-12-15 11:21:23.240618] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  72. [2016-12-15 11:21:23.240675] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  73. [2016-12-15 11:21:23.240700] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  74. [2016-12-15 11:21:23.240716] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  75. [2016-12-15 11:21:23.240732] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  76. [2016-12-15 11:21:23.240746] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  77. [2016-12-15 11:21:23.240761] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  78. [2016-12-15 11:21:23.240775] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  79. [2016-12-15 11:21:23.240791] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  80. [2016-12-15 11:21:23.240805] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  81. [2016-12-15 11:21:23.240820] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  82. [2016-12-15 11:21:23.240837] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31417: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  83. [2016-12-15 11:21:23.240903] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  84. [2016-12-15 11:21:23.241327] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  85. [2016-12-15 11:21:23.241366] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  86. [2016-12-15 11:21:23.241414] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  87. [2016-12-15 11:21:23.241431] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  88. [2016-12-15 11:21:23.241442] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  89. [2016-12-15 11:21:23.241451] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  90. [2016-12-15 11:21:23.241465] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  91. [2016-12-15 11:21:23.241475] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  92. [2016-12-15 11:21:23.241484] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  93. [2016-12-15 11:21:23.241494] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  94. [2016-12-15 11:21:23.241502] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  95. [2016-12-15 11:21:23.241511] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  96. [2016-12-15 11:21:23.241522] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31418: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  97. [2016-12-15 11:21:23.241572] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  98. [2016-12-15 11:21:23.241953] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x182fe) [0x7fef1376d2fe] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  99. [2016-12-15 11:21:23.241973] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: STATFS scheduled as fast fop
  100. [2016-12-15 11:21:23.242039] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c272) [0x7fef13781272] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  101. [2016-12-15 11:21:23.242186] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x1b59d) [0x7fef1377059d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  102. [2016-12-15 11:21:23.242210] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: WRITE scheduled as slow fop
  103. [2016-12-15 11:21:23.242387] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x268e6) [0x7fef1377b8e6] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  104. [2016-12-15 11:21:23.242455] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  105. [2016-12-15 11:21:23.242485] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  106. [2016-12-15 11:21:23.242536] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  107. [2016-12-15 11:21:23.242556] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  108. [2016-12-15 11:21:23.242570] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  109. [2016-12-15 11:21:23.242585] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  110. [2016-12-15 11:21:23.242599] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  111. [2016-12-15 11:21:23.242614] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  112. [2016-12-15 11:21:23.242630] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  113. [2016-12-15 11:21:23.242647] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  114. [2016-12-15 11:21:23.242661] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  115. [2016-12-15 11:21:23.242675] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  116. [2016-12-15 11:21:23.242693] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31420: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  117. [2016-12-15 11:21:23.242746] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  118. [2016-12-15 11:21:23.243095] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  119. [2016-12-15 11:21:23.243127] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  120. [2016-12-15 11:21:23.243173] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  121. [2016-12-15 11:21:23.243188] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  122. [2016-12-15 11:21:23.243203] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  123. [2016-12-15 11:21:23.243212] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  124. [2016-12-15 11:21:23.243221] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  125. [2016-12-15 11:21:23.243230] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  126. [2016-12-15 11:21:23.243239] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  127. [2016-12-15 11:21:23.243249] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  128. [2016-12-15 11:21:23.243258] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  129. [2016-12-15 11:21:23.243277] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  130. [2016-12-15 11:21:23.243289] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31421: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  131. [2016-12-15 11:21:23.243337] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  132. [2016-12-15 11:21:23.243733] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  133. [2016-12-15 11:21:23.243768] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  134. [2016-12-15 11:21:23.243838] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  135. [2016-12-15 11:21:23.243857] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  136. [2016-12-15 11:21:23.243868] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  137. [2016-12-15 11:21:23.243878] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  138. [2016-12-15 11:21:23.243887] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  139. [2016-12-15 11:21:23.243901] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  140. [2016-12-15 11:21:23.243911] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  141. [2016-12-15 11:21:23.243920] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  142. [2016-12-15 11:21:23.243929] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  143. [2016-12-15 11:21:23.243938] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  144. [2016-12-15 11:21:23.243949] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31422: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  145. [2016-12-15 11:21:23.244001] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  146. [2016-12-15 11:21:23.244359] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  147. [2016-12-15 11:21:23.244381] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x1b59d) [0x7fef1377059d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  148. [2016-12-15 11:21:23.244397] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: WRITE scheduled as slow fop
  149. [2016-12-15 11:21:23.244394] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  150. [2016-12-15 11:21:23.244444] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  151. [2016-12-15 11:21:23.244459] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  152. [2016-12-15 11:21:23.244469] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  153. [2016-12-15 11:21:23.244478] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  154. [2016-12-15 11:21:23.244486] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  155. [2016-12-15 11:21:23.244495] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  156. [2016-12-15 11:21:23.244508] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  157. [2016-12-15 11:21:23.244518] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  158. [2016-12-15 11:21:23.244527] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  159. [2016-12-15 11:21:23.244535] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  160. [2016-12-15 11:21:23.244546] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31423: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  161. [2016-12-15 11:21:23.244556] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x268e6) [0x7fef1377b8e6] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  162. [2016-12-15 11:21:23.244594] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  163. [2016-12-15 11:21:23.245070] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x3081d) [0x7fef1378581d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 2
  164. [2016-12-15 11:21:23.245095] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: LOOKUP scheduled as fast fop
  165. [2016-12-15 11:21:23.245136] D [MSGID: 0] [posix.c:257:posix_lookup] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-posix returned -1 error: No such file or directory [No such file or directory]
  166. [2016-12-15 11:21:23.245152] D [MSGID: 0] [changetimerecorder.c:307:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-changetimerecorder returned -1 error: No such file or directory [No such file or directory]
  167. [2016-12-15 11:21:23.245162] D [MSGID: 0] [bit-rot-stub.c:2710:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-bitrot-stub returned -1 error: No such file or directory [No such file or directory]
  168. [2016-12-15 11:21:23.245171] D [MSGID: 0] [posix-acl.c:917:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-access-control returned -1 error: No such file or directory [No such file or directory]
  169. [2016-12-15 11:21:23.245180] D [MSGID: 0] [posix.c:2649:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-locks returned -1 error: No such file or directory [No such file or directory]
  170. [2016-12-15 11:21:23.245190] D [MSGID: 0] [upcall.c:754:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-upcall returned -1 error: No such file or directory [No such file or directory]
  171. [2016-12-15 11:21:23.245198] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-io-threads returned -1 error: No such file or directory [No such file or directory]
  172. [2016-12-15 11:21:23.245212] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-marker returned -1 error: No such file or directory [No such file or directory]
  173. [2016-12-15 11:21:23.245221] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, glusterfs01-index returned -1 error: No such file or directory [No such file or directory]
  174. [2016-12-15 11:21:23.245230] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fef25ef606c, /var/disk2/brick returned -1 error: No such file or directory [No such file or directory]
  175. [2016-12-15 11:21:23.245241] D [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-glusterfs01-server: 31424: LOOKUP /db1/stack/02/13b5c202 (4dc916fe-c7d3-4541-b2cf-325b40d55d6e/13b5c202) ==> (No such file or directory) [No such file or directory]
  176. [2016-12-15 11:21:23.245289] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x2c961) [0x7fef13781961] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom126.example.com-2382-2016/12/14-01:05:23:315839-glusterfs01-client-3-0-9: ref-count 1
  177. [2016-12-15 11:21:23.245722] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x1b59d) [0x7fef1377059d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  178. [2016-12-15 11:21:23.245740] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-glusterfs01-io-threads: WRITE scheduled as slow fop
  179. [2016-12-15 11:21:23.245874] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x268e6) [0x7fef1377b8e6] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  180. [2016-12-15 11:21:23.246874] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x1b59d) [0x7fef1377059d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
  181. [2016-12-15 11:21:23.247019] D [client_t.c:412:gf_client_unref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x268e6) [0x7fef1377b8e6] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x9a5c) [0x7fef1375ea5c] -->/usr/lib64/libglusterfs.so.0(gf_client_unref+0x7f) [0x7fef2859232f] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 1
  182. [2016-12-15 11:21:23.248028] D [client_t.c:328:gf_client_ref] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x1b59d) [0x7fef1377059d] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/server.so(+0x16bea) [0x7fef1376bbea] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x6e) [0x7fef2859287e] ) 0-client_t: rickdom74.example.com-7393-2016/12/14-13:10:46:702212-glusterfs01-client-3-0-0: ref-count 2
RAW Paste Data Copied