valeriol

gluster.log.old

Mar 29th, 2021
37
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2021-03-28 03:30:18.812841] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-6: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  2. [2021-03-28 03:30:18.812908] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-14: Connected, attached to remote volume [{conn-name=MRIData-client-14}, {remote_subvol=/arbiter/1}]
  3. [2021-03-28 03:30:18.812982] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-7: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  4. [2021-03-28 03:30:18.813028] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-9: changing port to 49152 (from 0)
  5. [2021-03-28 03:30:18.813039] I [socket.c:849:__socket_shutdown] 0-MRIData-client-9: intentional socket shutdown(27)
  6. [2021-03-28 03:30:18.813075] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-15: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  7. [2021-03-28 03:30:18.813647] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-16: changing port to 49156 (from 0)
  8. [2021-03-28 03:30:18.813663] I [socket.c:849:__socket_shutdown] 0-MRIData-client-16: intentional socket shutdown(22)
  9. [2021-03-28 03:30:18.813683] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-10: changing port to 49153 (from 0)
  10. [2021-03-28 03:30:18.813693] I [socket.c:849:__socket_shutdown] 0-MRIData-client-10: intentional socket shutdown(23)
  11. [2021-03-28 03:30:18.813701] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-11: changing port to 49154 (from 0)
  12. [2021-03-28 03:30:18.813711] I [socket.c:849:__socket_shutdown] 0-MRIData-client-11: intentional socket shutdown(14)
  13. [2021-03-28 03:30:18.813751] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-6: Connected, attached to remote volume [{conn-name=MRIData-client-6}, {remote_subvol=/gluster1/data}]
  14. [2021-03-28 03:30:18.813765] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-3: Subvolume 'MRIData-client-6' came back up; going online.
  15. [2021-03-28 03:30:18.813977] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-17: changing port to 49155 (from 0)
  16. [2021-03-28 03:30:18.813992] I [socket.c:849:__socket_shutdown] 0-MRIData-client-17: intentional socket shutdown(31)
  17. [2021-03-28 03:30:18.815724] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-7: Connected, attached to remote volume [{conn-name=MRIData-client-7}, {remote_subvol=/gluster2/data}]
  18. [2021-03-28 03:30:18.815767] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-3: Client-quorum is met
  19. [2021-03-28 03:30:18.815905] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-15: Connected, attached to remote volume [{conn-name=MRIData-client-15}, {remote_subvol=/arbiter/1}]
  20. [2021-03-28 03:30:18.816959] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-8: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  21. [2021-03-28 03:30:18.817150] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-9: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  22. [2021-03-28 03:30:18.817193] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-16: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  23. [2021-03-28 03:30:18.817221] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-11: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  24. [2021-03-28 03:30:18.817253] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-10: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  25. [2021-03-28 03:30:18.817841] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-8: Connected, attached to remote volume [{conn-name=MRIData-client-8}, {remote_subvol=/gluster3/data}]
  26. [2021-03-28 03:30:18.817849] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-4: Subvolume 'MRIData-client-8' came back up; going online.
  27. [2021-03-28 03:30:18.817960] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-10: Connected, attached to remote volume [{conn-name=MRIData-client-10}, {remote_subvol=/gluster2/data}]
  28. [2021-03-28 03:30:18.818065] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-5: Subvolume 'MRIData-client-10' came back up; going online.
  29. [2021-03-28 03:30:18.818140] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-9: Connected, attached to remote volume [{conn-name=MRIData-client-9}, {remote_subvol=/gluster1/data}]
  30. [2021-03-28 03:30:18.818285] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-16: Connected, attached to remote volume [{conn-name=MRIData-client-16}, {remote_subvol=/arbiter/2}]
  31. [2021-03-28 03:30:18.818348] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-4: Client-quorum is met
  32. [2021-03-28 03:30:18.818464] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-11: Connected, attached to remote volume [{conn-name=MRIData-client-11}, {remote_subvol=/gluster3/data}]
  33. [2021-03-28 03:30:18.818579] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-17: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  34. [2021-03-28 03:30:18.818653] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-5: Client-quorum is met
  35. [2021-03-28 03:30:18.819471] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-17: Connected, attached to remote volume [{conn-name=MRIData-client-17}, {remote_subvol=/arbiter/1}]
  36. [2021-03-28 03:30:18.820529] I [fuse-bridge.c:5300:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
  37. [2021-03-28 03:30:18.820545] I [fuse-bridge.c:5928:fuse_graph_sync] 0-fuse: switched to graph 0
  38. [2021-03-28 03:30:25.154629] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x715ac) [0x7f17741335ac] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x691c) [0x7f176fdd491c] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f176fdd45bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for 676666a6-b81f-48da-9359-2e2260b253c5
  39. [2021-03-28 03:30:25.154657] W [fuse-bridge.c:1704:fuse_setattr_cbk] 0-glusterfs-fuse: 146: SETATTR() /fff => -1 (Invalid argument)
  40. [2021-03-28 03:30:30.681479] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f1774140b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f176fdd4767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f176fdd45bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for 676666a6-b81f-48da-9359-2e2260b253c5
  41. [2021-03-28 03:30:30.681500] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 182: STAT() /fff => -1 (Invalid argument)
  42. [2021-03-28 03:30:36.705628] I [MSGID: 133022] [shard.c:3768:shard_delete_shards] 0-MRIData-shard: Deleted shards of gfid=676666a6-b81f-48da-9359-2e2260b253c5 from backend
  43. [2021-03-28 05:28:01.599868] I [fuse-bridge.c:6237:fuse_thread_proc] 0-fuse: initiating unmount of /gluster
  44. [2021-03-28 05:28:01.600079] W [glusterfsd.c:1439:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x814a) [0x7f177be2014a] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x563417590b5d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x5634175909a8] ) 0-: received signum (15), shutting down
  45. [2021-03-28 05:28:01.600099] I [fuse-bridge.c:7069:fini] 0-fuse: Unmounting '/gluster'.
  46. [2021-03-28 05:28:01.600120] I [fuse-bridge.c:7074:fini] 0-fuse: Closing fuse connection to '/gluster'.
  47. [2021-03-28 05:28:36.533521] I [MSGID: 100030] [glusterfsd.c:2689:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=8.4}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=hydra1 --volfile-id=/MRIData /gluster}]
  48. [2021-03-28 05:28:36.534453] I [glusterfsd.c:2424:daemonize] 0-glusterfs: Pid of current running process is 104056
  49. [2021-03-28 05:28:36.539663] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
  50. [2021-03-28 05:28:36.539716] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
  51. [2021-03-28 05:28:36.551647] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-0: parent translators are ready, attempting connect on transport []
  52. [2021-03-28 05:28:36.553256] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-1: parent translators are ready, attempting connect on transport []
  53. [2021-03-28 05:28:36.553848] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-0: changing port to 49152 (from 0)
  54. [2021-03-28 05:28:36.553866] I [socket.c:849:__socket_shutdown] 0-MRIData-client-0: intentional socket shutdown(13)
  55. [2021-03-28 05:28:36.554799] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-12: parent translators are ready, attempting connect on transport []
  56. [2021-03-28 05:28:36.555761] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-1: changing port to 49153 (from 0)
  57. [2021-03-28 05:28:36.555781] I [socket.c:849:__socket_shutdown] 0-MRIData-client-1: intentional socket shutdown(14)
  58. [2021-03-28 05:28:36.556285] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-2: parent translators are ready, attempting connect on transport []
  59. [2021-03-28 05:28:36.557272] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  60. [2021-03-28 05:28:36.557684] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-12: changing port to 49154 (from 0)
  61. [2021-03-28 05:28:36.557698] I [socket.c:849:__socket_shutdown] 0-MRIData-client-12: intentional socket shutdown(16)
  62. [2021-03-28 05:28:36.557713] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-1: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  63. [2021-03-28 05:28:36.557773] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-3: parent translators are ready, attempting connect on transport []
  64. [2021-03-28 05:28:36.559241] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-13: parent translators are ready, attempting connect on transport []
  65. [2021-03-28 05:28:36.559267] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-0: Connected, attached to remote volume [{conn-name=MRIData-client-0}, {remote_subvol=/gluster1/data}]
  66. [2021-03-28 05:28:36.559278] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-0: Subvolume 'MRIData-client-0' came back up; going online.
  67. [2021-03-28 05:28:36.559813] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-1: Connected, attached to remote volume [{conn-name=MRIData-client-1}, {remote_subvol=/gluster2/data}]
  68. [2021-03-28 05:28:36.559891] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-0: Client-quorum is met
  69. [2021-03-28 05:28:36.560215] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-2: changing port to 49155 (from 0)
  70. [2021-03-28 05:28:36.560228] I [socket.c:849:__socket_shutdown] 0-MRIData-client-2: intentional socket shutdown(14)
  71. [2021-03-28 05:28:36.560546] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-4: parent translators are ready, attempting connect on transport []
  72. [2021-03-28 05:28:36.561525] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-12: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  73. [2021-03-28 05:28:36.561818] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-3: changing port to 49152 (from 0)
  74. [2021-03-28 05:28:36.561833] I [socket.c:849:__socket_shutdown] 0-MRIData-client-3: intentional socket shutdown(16)
  75. [2021-03-28 05:28:36.561833] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-5: parent translators are ready, attempting connect on transport []
  76. [2021-03-28 05:28:36.563119] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-14: parent translators are ready, attempting connect on transport []
  77. [2021-03-28 05:28:36.563159] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-13: changing port to 49156 (from 0)
  78. [2021-03-28 05:28:36.563170] I [socket.c:849:__socket_shutdown] 0-MRIData-client-13: intentional socket shutdown(17)
  79. [2021-03-28 05:28:36.563184] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-2: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  80. [2021-03-28 05:28:36.563249] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-12: Connected, attached to remote volume [{conn-name=MRIData-client-12}, {remote_subvol=/arbiter/1}]
  81. [2021-03-28 05:28:36.564380] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-6: parent translators are ready, attempting connect on transport []
  82. [2021-03-28 05:28:36.565588] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-2: Connected, attached to remote volume [{conn-name=MRIData-client-2}, {remote_subvol=/gluster3/data}]
  83. [2021-03-28 05:28:36.565600] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-1: Subvolume 'MRIData-client-2' came back up; going online.
  84. [2021-03-28 05:28:36.565669] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-7: parent translators are ready, attempting connect on transport []
  85. [2021-03-28 05:28:36.565769] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-3: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  86. [2021-03-28 05:28:36.565867] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-4: changing port to 49153 (from 0)
  87. [2021-03-28 05:28:36.565882] I [socket.c:849:__socket_shutdown] 0-MRIData-client-4: intentional socket shutdown(14)
  88. [2021-03-28 05:28:36.566988] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-15: parent translators are ready, attempting connect on transport []
  89. [2021-03-28 05:28:36.567264] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-5: changing port to 49154 (from 0)
  90. [2021-03-28 05:28:36.567277] I [socket.c:849:__socket_shutdown] 0-MRIData-client-5: intentional socket shutdown(20)
  91. [2021-03-28 05:28:36.567289] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-13: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  92. [2021-03-28 05:28:36.567322] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-14: changing port to 49155 (from 0)
  93. [2021-03-28 05:28:36.567333] I [socket.c:849:__socket_shutdown] 0-MRIData-client-14: intentional socket shutdown(16)
  94. [2021-03-28 05:28:36.567347] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-3: Connected, attached to remote volume [{conn-name=MRIData-client-3}, {remote_subvol=/gluster1/data}]
  95. [2021-03-28 05:28:36.568226] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-8: parent translators are ready, attempting connect on transport []
  96. [2021-03-28 05:28:36.568255] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-1: Client-quorum is met
  97. [2021-03-28 05:28:36.569530] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-9: parent translators are ready, attempting connect on transport []
  98. [2021-03-28 05:28:36.570787] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-16: parent translators are ready, attempting connect on transport []
  99. [2021-03-28 05:28:36.571087] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-6: changing port to 49152 (from 0)
  100. [2021-03-28 05:28:36.571101] I [socket.c:849:__socket_shutdown] 0-MRIData-client-6: intentional socket shutdown(17)
  101. [2021-03-28 05:28:36.571137] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-13: Connected, attached to remote volume [{conn-name=MRIData-client-13}, {remote_subvol=/arbiter/2}]
  102. [2021-03-28 05:28:36.571265] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-4: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  103. [2021-03-28 05:28:36.572071] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-10: parent translators are ready, attempting connect on transport []
  104. [2021-03-28 05:28:36.572720] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-5: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  105. [2021-03-28 05:28:36.572785] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-7: changing port to 49153 (from 0)
  106. [2021-03-28 05:28:36.572809] I [socket.c:849:__socket_shutdown] 0-MRIData-client-7: intentional socket shutdown(23)
  107. [2021-03-28 05:28:36.572894] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-4: Connected, attached to remote volume [{conn-name=MRIData-client-4}, {remote_subvol=/gluster2/data}]
  108. [2021-03-28 05:28:36.572900] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-2: Subvolume 'MRIData-client-4' came back up; going online.
  109. [2021-03-28 05:28:36.573356] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-11: parent translators are ready, attempting connect on transport []
  110. [2021-03-28 05:28:36.574492] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-8: changing port to 49154 (from 0)
  111. [2021-03-28 05:28:36.574508] I [socket.c:849:__socket_shutdown] 0-MRIData-client-8: intentional socket shutdown(25)
  112. [2021-03-28 05:28:36.574524] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-14: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  113. [2021-03-28 05:28:36.574552] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-9: changing port to 49152 (from 0)
  114. [2021-03-28 05:28:36.574563] I [socket.c:849:__socket_shutdown] 0-MRIData-client-9: intentional socket shutdown(27)
  115. [2021-03-28 05:28:36.574574] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-6: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  116. [2021-03-28 05:28:36.574606] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-5: Connected, attached to remote volume [{conn-name=MRIData-client-5}, {remote_subvol=/gluster3/data}]
  117. [2021-03-28 05:28:36.574649] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-17: parent translators are ready, attempting connect on transport []
  118. [2021-03-28 05:28:36.574706] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-2: Client-quorum is met
  119. [2021-03-28 05:28:36.574888] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-15: changing port to 49155 (from 0)
  120. [2021-03-28 05:28:36.574900] I [socket.c:849:__socket_shutdown] 0-MRIData-client-15: intentional socket shutdown(14)
  121. Final graph:
  122. +------------------------------------------------------------------------------+
  123. 1: volume MRIData-client-0
  124. 2: type protocol/client
  125. 3: option opversion 80000
  126. 4: option clnt-lk-version 1
  127. 5: option volfile-checksum 0
  128. 6: option volfile-key /MRIData
  129. 7: option client-version 8.4
  130. 8: option process-name fuse
  131. 9: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-0-RECON_NO:-0
  132. 10: option fops-version 1298437
  133. 11: option ping-timeout 10
  134. 12: option remote-host hydra1
  135. 13: option remote-subvolume /gluster1/data
  136. 14: option transport-type socket
  137. 15: option transport.address-family inet
  138. 16: option filter-O_DIRECT enable
  139. 17: option transport.tcp-user-timeout 0
  140. 18: option transport.socket.keepalive-time 20
  141. 19: option transport.socket.keepalive-interval 2
  142. 20: option transport.socket.keepalive-count 9
  143. 21: option send-gids true
  144. 22: end-volume
  145. 23:
  146. 24: volume MRIData-client-1
  147. 25: type protocol/client
  148. 26: option opversion 80000
  149. 27: option clnt-lk-version 1
  150. 28: option volfile-checksum 0
  151. 29: option volfile-key /MRIData
  152. 30: option client-version 8.4
  153. 31: option process-name fuse
  154. 32: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-1-RECON_NO:-0
  155. 33: option fops-version 1298437
  156. 34: option ping-timeout 10
  157. 35: option remote-host hydra1
  158. 36: option remote-subvolume /gluster2/data
  159. 37: option transport-type socket
  160. 38: option transport.address-family inet
  161. 39: option filter-O_DIRECT enable
  162. 40: option transport.tcp-user-timeout 0
  163. 41: option transport.socket.keepalive-time 20
  164. 42: option transport.socket.keepalive-interval 2
  165. 43: option transport.socket.keepalive-count 9
  166. 44: option send-gids true
  167. 45: end-volume
  168. 46:
  169. 47: volume MRIData-client-12
  170. 48: type protocol/client
  171. 49: option opversion 80000
  172. 50: option clnt-lk-version 1
  173. 51: option volfile-checksum 0
  174. 52: option volfile-key /MRIData
  175. 53: option client-version 8.4
  176. 54: option process-name fuse
  177. 55: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-12-RECON_NO:-0
  178. 56: option fops-version 1298437
  179. 57: option ping-timeout 10
  180. 58: option remote-host hydra1
  181. 59: option remote-subvolume /arbiter/1
  182. 60: option transport-type socket
  183. 61: option transport.address-family inet
  184. 62: option filter-O_DIRECT enable
  185. 63: option transport.tcp-user-timeout 0
  186. 64: option transport.socket.keepalive-time 20
  187. 65: option transport.socket.keepalive-interval 2
  188. 66: option transport.socket.keepalive-count 9
  189. 67: option send-gids true
  190. 68: end-volume
  191. 69:
  192. 70: volume MRIData-replicate-0
  193. 71: type cluster/replicate
  194. 72: option afr-pending-xattr MRIData-client-0,MRIData-client-1,MRIData-client-12
  195. 73: option arbiter-count 1
  196. 74: option metadata-self-heal on
  197. 75: option data-self-heal on
  198. 76: option entry-self-heal on
  199. 77: option data-self-heal-algorithm full
  200. 78: option eager-lock enable
  201. 79: option quorum-type auto
  202. 80: option choose-local off
  203. 81: option shd-max-threads 8
  204. 82: option shd-wait-qlength 10000
  205. 83: option locking-scheme granular
  206. 84: option use-compound-fops off
  207. 85: subvolumes MRIData-client-0 MRIData-client-1 MRIData-client-12
  208. 86: end-volume
  209. 87:
  210. 88: volume MRIData-client-2
  211. 89: type protocol/client
  212. 90: option opversion 80000
  213. 91: option clnt-lk-version 1
  214. 92: option volfile-checksum 0
  215. 93: option volfile-key /MRIData
  216. 94: option client-version 8.4
  217. 95: option process-name fuse
  218. 96: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-2-RECON_NO:-0
  219. 97: option fops-version 1298437
  220. 98: option ping-timeout 10
  221. 99: option remote-host hydra1
  222. 100: option remote-subvolume /gluster3/data
  223. 101: option transport-type socket
  224. 102: option transport.address-family inet
  225. 103: option filter-O_DIRECT enable
  226. 104: option transport.tcp-user-timeout 0
  227. 105: option transport.socket.keepalive-time 20
  228. 106: option transport.socket.keepalive-interval 2
  229. 107: option transport.socket.keepalive-count 9
  230. 108: option send-gids true
  231. 109: end-volume
  232. 110:
  233. 111: volume MRIData-client-3
  234. 112: type protocol/client
  235. 113: option opversion 80000
  236. 114: option clnt-lk-version 1
  237. 115: option volfile-checksum 0
  238. 116: option volfile-key /MRIData
  239. 117: option client-version 8.4
  240. 118: option process-name fuse
  241. 119: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-3-RECON_NO:-0
  242. 120: option fops-version 1298437
  243. 121: option ping-timeout 10
  244. 122: option remote-host hydra2
  245. 123: option remote-subvolume /gluster1/data
  246. 124: option transport-type socket
  247. 125: option transport.address-family inet
  248. 126: option filter-O_DIRECT enable
  249. 127: option transport.tcp-user-timeout 0
  250. 128: option transport.socket.keepalive-time 20
  251. 129: option transport.socket.keepalive-interval 2
  252. 130: option transport.socket.keepalive-count 9
  253. 131: option send-gids true
  254. 132: end-volume
  255. 133:
  256. 134: volume MRIData-client-13
  257. 135: type protocol/client
  258. 136: option opversion 80000
  259. 137: option clnt-lk-version 1
  260. 138: option volfile-checksum 0
  261. 139: option volfile-key /MRIData
  262. 140: option client-version 8.4
  263. 141: option process-name fuse
  264. 142: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-13-RECON_NO:-0
  265. 143: option fops-version 1298437
  266. 144: option ping-timeout 10
  267. 145: option remote-host hydra1
  268. 146: option remote-subvolume /arbiter/2
  269. 147: option transport-type socket
  270. 148: option transport.address-family inet
  271. 149: option filter-O_DIRECT enable
  272. 150: option transport.tcp-user-timeout 0
  273. 151: option transport.socket.keepalive-time 20
  274. 152: option transport.socket.keepalive-interval 2
  275. 153: option transport.socket.keepalive-count 9
  276. 154: option send-gids true
  277. 155: end-volume
  278. 156:
  279. 157: volume MRIData-replicate-1
  280. 158: type cluster/replicate
  281. 159: option afr-pending-xattr MRIData-client-2,MRIData-client-3,MRIData-client-13
  282. 160: option arbiter-count 1
  283. 161: option metadata-self-heal on
  284. 162: option data-self-heal on
  285. 163: option entry-self-heal on
  286. 164: option data-self-heal-algorithm full
  287. 165: option eager-lock enable
  288. 166: option quorum-type auto
  289. 167: option choose-local off
  290. 168: option shd-max-threads 8
  291. 169: option shd-wait-qlength 10000
  292. 170: option locking-scheme granular
  293. 171: option use-compound-fops off
  294. 172: subvolumes MRIData-client-2 MRIData-client-3 MRIData-client-13
  295. 173: end-volume
  296. 174:
  297. 175: volume MRIData-client-4
  298. 176: type protocol/client
  299. 177: option opversion 80000
  300. 178: option clnt-lk-version 1
  301. 179: option volfile-checksum 0
  302. 180: option volfile-key /MRIData
  303. 181: option client-version 8.4
  304. 182: option process-name fuse
  305. 183: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-4-RECON_NO:-0
  306. 184: option fops-version 1298437
  307. 185: option ping-timeout 10
  308. 186: option remote-host hydra2
  309. 187: option remote-subvolume /gluster2/data
  310. 188: option transport-type socket
  311. 189: option transport.address-family inet
  312. 190: option filter-O_DIRECT enable
  313. 191: option transport.tcp-user-timeout 0
  314. 192: option transport.socket.keepalive-time 20
  315. 193: option transport.socket.keepalive-interval 2
  316. 194: option transport.socket.keepalive-count 9
  317. 195: option send-gids true
  318. 196: end-volume
  319. 197:
  320. 198: volume MRIData-client-5
  321. 199: type protocol/client
  322. 200: option opversion 80000
  323. 201: option clnt-lk-version 1
  324. 202: option volfile-checksum 0
  325. 203: option volfile-key /MRIData
  326. 204: option client-version 8.4
  327. 205: option process-name fuse
  328. 206: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-5-RECON_NO:-0
  329. 207: option fops-version 1298437
  330. 208: option ping-timeout 10
  331. 209: option remote-host hydra2
  332. 210: option remote-subvolume /gluster3/data
  333. 211: option transport-type socket
  334. 212: option transport.address-family inet
  335. 213: option filter-O_DIRECT enable
  336. 214: option transport.tcp-user-timeout 0
  337. 215: option transport.socket.keepalive-time 20
  338. 216: option transport.socket.keepalive-interval 2
  339. 217: option transport.socket.keepalive-count 9
  340. 218: option send-gids true
  341. 219: end-volume
  342. 220:
  343. 221: volume MRIData-client-14
  344. 222: type protocol/client
  345. 223: option opversion 80000
  346. 224: option clnt-lk-version 1
  347. 225: option volfile-checksum 0
  348. 226: option volfile-key /MRIData
  349. 227: option client-version 8.4
  350. 228: option process-name fuse
  351. 229: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-14-RECON_NO:-0
  352. 230: option fops-version 1298437
  353. 231: option ping-timeout 10
  354. 232: option remote-host hydra2
  355. 233: option remote-subvolume /arbiter/1
  356. 234: option transport-type socket
  357. 235: option transport.address-family inet
  358. 236: option filter-O_DIRECT enable
  359. 237: option transport.tcp-user-timeout 0
  360. 238: option transport.socket.keepalive-time 20
  361. 239: option transport.socket.keepalive-interval 2
  362. 240: option transport.socket.keepalive-count 9
  363. 241: option send-gids true
  364. 242: end-volume
  365. 243:
  366. 244: volume MRIData-replicate-2
  367. 245: type cluster/replicate
  368. 246: option afr-pending-xattr MRIData-client-4,MRIData-client-5,MRIData-client-14
  369. 247: option arbiter-count 1
  370. 248: option metadata-self-heal on
  371. 249: option data-self-heal on
  372. 250: option entry-self-heal on
  373. 251: option data-self-heal-algorithm full
  374. 252: option eager-lock enable
  375. 253: option quorum-type auto
  376. 254: option choose-local off
  377. 255: option shd-max-threads 8
  378. 256: option shd-wait-qlength 10000
  379. 257: option locking-scheme granular
  380. 258: option use-compound-fops off
  381. 259: subvolumes MRIData-client-4 MRIData-client-5 MRIData-client-14
  382. 260: end-volume
  383. 261:
  384. 262: volume MRIData-client-6
  385. 263: type protocol/client
  386. 264: option opversion 80000
  387. 265: option clnt-lk-version 1
  388. 266: option volfile-checksum 0
  389. 267: option volfile-key /MRIData
  390. 268: option client-version 8.4
  391. 269: option process-name fuse
  392. 270: option process-uuid CTX_ID:5bae3ff1-7707-4295-8043-cfc3ace80764-GRAPH_ID:0-PID:104056-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-6-RECON_NO:-0
  393. 271: option fops-version 1298437
  394. 272: option ping-timeout 10
  395. 273: option remote-host hydra3
  396. 274: option remote-subvolume /gluster1/data
  397. 275: option transport-type socket
  398. 276: option transport.address-family inet
  399. 277: option filter-O_DIRECT enable
  400. 278: option transport.tcp-user-timeout 0
  401. 279: option transport.socket.keepalive-time 20
  402. 280: option transport.socket.keepalive-interval 2
  403. 281: option transport.socket.keepalive-count 9
  404. 282: option send-gids true
  405. 283: end-volume
  406. 284:
  407. 285: volume MRIData-client-7
  408. 286: type protocol/client
  409. 287: option ping-timeout 10
  410. 288: option remote-host hydra3
  411. 289: option remote-subvolume /gluster2/data
  412. 290: option transport-type socket
  413. 291: option transport.address-family inet
  414. 292: option filter-O_DIRECT enable
  415. 293: option transport.tcp-user-timeout 0
  416. 294: option transport.socket.keepalive-time 20
  417. 295: option transport.socket.keepalive-interval 2
  418. 296: option transport.socket.keepalive-count 9
  419. 297: option send-gids true
  420. 298: end-volume
  421. 299:
  422. 300: volume MRIData-client-15
  423. 301: type protocol/client
  424. 302: option ping-timeout 10
  425. 303: option remote-host hydra3
  426. 304: option remote-subvolume /arbiter/1
  427. 305: option transport-type socket
  428. 306: option transport.address-family inet
  429. 307: option filter-O_DIRECT enable
  430. 308: option transport.tcp-user-timeout 0
  431. 309: option transport.socket.keepalive-time 20
  432. 310: option transport.socket.keepalive-interval 2
  433. 311: option transport.socket.keepalive-count 9
  434. 312: option send-gids true
  435. 313: end-volume
  436. 314:
  437. 315: volume MRIData-replicate-3
  438. 316: type cluster/replicate
  439. 317: option afr-pending-xattr MRIData-client-6,MRIData-client-7,MRIData-client-15
  440. 318: option arbiter-count 1
  441. 319: option metadata-self-heal on
  442. 320: option data-self-heal on
  443. 321: option entry-self-heal on
  444. 322: option data-self-heal-algorithm full
  445. 323: option eager-lock enable
  446. 324: option quorum-type auto
  447. 325: option choose-local off
  448. 326: option shd-max-threads 8
  449. 327: option shd-wait-qlength 10000
  450. 328: option locking-scheme granular
  451. 329: option use-compound-fops off
  452. 330: subvolumes MRIData-client-6 MRIData-client-7 MRIData-client-15
  453. 331: end-volume
  454. 332:
  455. 333: volume MRIData-client-8
  456. 334: type protocol/client
  457. 335: option ping-timeout 10
  458. 336: option remote-host hydra3
  459. 337: option remote-subvolume /gluster3/data
  460. 338: option transport-type socket
  461. 339: option transport.address-family inet
  462. 340: option filter-O_DIRECT enable
  463. 341: option transport.tcp-user-timeout 0
  464. 342: option transport.socket.keepalive-time 20
  465. 343: option transport.socket.keepalive-interval 2
  466. 344: option transport.socket.keepalive-count 9
  467. 345: option send-gids true
  468. 346: end-volume
  469. 347:
  470. 348: volume MRIData-client-9
  471. 349: type protocol/client
  472. 350: option ping-timeout 10
  473. 351: option remote-host hydra4
  474. 352: option remote-subvolume /gluster1/data
  475. 353: option transport-type socket
  476. 354: option transport.address-family inet
  477. 355: option filter-O_DIRECT enable
  478. 356: option transport.tcp-user-timeout 0
  479. 357: option transport.socket.keepalive-time 20
  480. 358: option transport.socket.keepalive-interval 2
  481. 359: option transport.socket.keepalive-count 9
  482. 360: option send-gids true
  483. 361: end-volume
  484. 362:
  485. 363: volume MRIData-client-16
  486. 364: type protocol/client
  487. 365: option ping-timeout 10
  488. 366: option remote-host hydra3
  489. 367: option remote-subvolume /arbiter/2
  490. 368: option transport-type socket
  491. 369: option transport.address-family inet
  492. 370: option filter-O_DIRECT enable
  493. 371: option transport.tcp-user-timeout 0
  494. 372: option transport.socket.keepalive-time 20
  495. 373: option transport.socket.keepalive-interval 2
  496. 374: option transport.socket.keepalive-count 9
  497. 375: option send-gids true
  498. 376: end-volume
  499. 377:
  500. 378: volume MRIData-replicate-4
  501. 379: type cluster/replicate
  502. 380: option afr-pending-xattr MRIData-client-8,MRIData-client-9,MRIData-client-16
  503. 381: option arbiter-count 1
  504. 382: option metadata-self-heal on
  505. 383: option data-self-heal on
  506. 384: option entry-self-heal on
  507. 385: option data-self-heal-algorithm full
  508. 386: option eager-lock enable
  509. 387: option quorum-type auto
  510. 388: option choose-local off
  511. 389: option shd-max-threads 8
  512. 390: option shd-wait-qlength 10000
  513. 391: option locking-scheme granular
  514. 392: option use-compound-fops off
  515. 393: subvolumes MRIData-client-8 MRIData-client-9 MRIData-client-16
  516. 394: end-volume
  517. 395:
  518. 396: volume MRIData-client-10
  519. 397: type protocol/client
  520. 398: option ping-timeout 10
  521. 399: option remote-host hydra4
  522. 400: option remote-subvolume /gluster2/data
  523. 401: option transport-type socket
  524. 402: option transport.address-family inet
  525. 403: option filter-O_DIRECT enable
  526. 404: option transport.tcp-user-timeout 0
  527. 405: option transport.socket.keepalive-time 20
  528. 406: option transport.socket.keepalive-interval 2
  529. 407: option transport.socket.keepalive-count 9
  530. 408: option send-gids true
  531. 409: end-volume
  532. 410:
  533. 411: volume MRIData-client-11
  534. 412: type protocol/client
  535. 413: option ping-timeout 10
  536. 414: option remote-host hydra4
  537. 415: option remote-subvolume /gluster3/data
  538. 416: option transport-type socket
  539. 417: option transport.address-family inet
  540. 418: option filter-O_DIRECT enable
  541. 419: option transport.tcp-user-timeout 0
  542. 420: option transport.socket.keepalive-time 20
  543. 421: option transport.socket.keepalive-interval 2
  544. 422: option transport.socket.keepalive-count 9
  545. 423: option send-gids true
  546. 424: end-volume
  547. 425:
  548. 426: volume MRIData-client-17
  549. 427: type protocol/client
  550. 428: option ping-timeout 10
  551. 429: option remote-host hydra4
  552. 430: option remote-subvolume /arbiter/1
  553. 431: option transport-type socket
  554. 432: option transport.address-family inet
  555. 433: option filter-O_DIRECT enable
  556. 434: option transport.tcp-user-timeout 0
  557. 435: option transport.socket.keepalive-time 20
  558. 436: option transport.socket.keepalive-interval 2
  559. 437: option transport.socket.keepalive-count 9
  560. 438: option send-gids true
  561. 439: end-volume
  562. 440:
  563. 441: volume MRIData-replicate-5
  564. 442: type cluster/replicate
  565. 443: option afr-pending-xattr MRIData-client-10,MRIData-client-11,MRIData-client-17
  566. 444: option arbiter-count 1
  567. 445: option metadata-self-heal on
  568. 446: option data-self-heal on
  569. 447: option entry-self-heal on
  570. 448: option data-self-heal-algorithm full
  571. 449: option eager-lock enable
  572. 450: option quorum-type auto
  573. 451: option choose-local off
  574. 452: option shd-max-threads 8
  575. 453: option shd-wait-qlength 10000
  576. 454: option locking-scheme granular
  577. 455: option use-compound-fops off
  578. 456: subvolumes MRIData-client-10 MRIData-client-11 MRIData-client-17
  579. 457: end-volume
  580. 458:
  581. 459: volume MRIData-dht
  582. 460: type cluster/distribute
  583. 461: option lock-migration off
  584. 462: option force-migration off
  585. 463: subvolumes MRIData-replicate-0 MRIData-replicate-1 MRIData-replicate-2 MRIData-replicate-3 MRIData-replicate-4 MRIData-replicate-5
  586. 464: end-volume
  587. 465:
  588. 466: volume MRIData-shard
  589. 467: type features/shard
  590. 468: subvolumes MRIData-dht
  591. 469: end-volume
  592. 470:
  593. 471: volume MRIData-write-behind
  594. 472: type performance/write-behind
  595. 473: subvolumes MRIData-shard
  596. 474: end-volume
  597. 475:
  598. 476: volume MRIData-readdir-ahead
  599. 477: type performance/readdir-ahead
  600. 478: option parallel-readdir off
  601. 479: option rda-request-size 131072
  602. 480: option rda-cache-limit 10MB
  603. 481: subvolumes MRIData-write-behind
  604. 482: end-volume
  605. 483:
  606. 484: volume MRIData-open-behind
  607. 485: type performance/open-behind
  608. 486: subvolumes MRIData-readdir-ahead
  609. 487: end-volume
  610. 488:
  611. 489: volume MRIData-md-cache
  612. 490: type performance/md-cache
  613. 491: subvolumes MRIData-open-behind
  614. 492: end-volume
  615. 493:
  616. 494: volume MRIData
  617. 495: type debug/io-stats
  618. 496: option log-level INFO
  619. 497: option latency-measurement off
  620. 498: option count-fop-hits off
  621. 499: subvolumes MRIData-md-cache
  622. 500: end-volume
  623. 501:
  624. 502: volume meta-autoload
  625. 503: type meta
  626. 504: subvolumes MRIData
  627. 505: end-volume
  628. 506:
  629. +------------------------------------------------------------------------------+
  630. [2021-03-28 05:28:36.577523] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-16: changing port to 49156 (from 0)
  631. [2021-03-28 05:28:36.577541] I [socket.c:849:__socket_shutdown] 0-MRIData-client-16: intentional socket shutdown(28)
  632. [2021-03-28 05:28:36.577580] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-14: Connected, attached to remote volume [{conn-name=MRIData-client-14}, {remote_subvol=/arbiter/1}]
  633. [2021-03-28 05:28:36.577703] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-7: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  634. [2021-03-28 05:28:36.577765] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-6: Connected, attached to remote volume [{conn-name=MRIData-client-6}, {remote_subvol=/gluster1/data}]
  635. [2021-03-28 05:28:36.577772] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-3: Subvolume 'MRIData-client-6' came back up; going online.
  636. [2021-03-28 05:28:36.578065] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-8: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  637. [2021-03-28 05:28:36.578108] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-10: changing port to 49153 (from 0)
  638. [2021-03-28 05:28:36.578120] I [socket.c:849:__socket_shutdown] 0-MRIData-client-10: intentional socket shutdown(29)
  639. [2021-03-28 05:28:36.579276] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-17: changing port to 49155 (from 0)
  640. [2021-03-28 05:28:36.579294] I [socket.c:849:__socket_shutdown] 0-MRIData-client-17: intentional socket shutdown(30)
  641. [2021-03-28 05:28:36.579309] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-15: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  642. [2021-03-28 05:28:36.579339] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-11: changing port to 49154 (from 0)
  643. [2021-03-28 05:28:36.579349] I [socket.c:849:__socket_shutdown] 0-MRIData-client-11: intentional socket shutdown(23)
  644. [2021-03-28 05:28:36.579361] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-9: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  645. [2021-03-28 05:28:36.579392] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-7: Connected, attached to remote volume [{conn-name=MRIData-client-7}, {remote_subvol=/gluster2/data}]
  646. [2021-03-28 05:28:36.579485] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-3: Client-quorum is met
  647. [2021-03-28 05:28:36.579502] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-8: Connected, attached to remote volume [{conn-name=MRIData-client-8}, {remote_subvol=/gluster3/data}]
  648. [2021-03-28 05:28:36.579924] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-4: Subvolume 'MRIData-client-8' came back up; going online.
  649. [2021-03-28 05:28:36.581277] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-9: Connected, attached to remote volume [{conn-name=MRIData-client-9}, {remote_subvol=/gluster1/data}]
  650. [2021-03-28 05:28:36.581373] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-4: Client-quorum is met
  651. [2021-03-28 05:28:36.581442] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-16: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  652. [2021-03-28 05:28:36.581485] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-10: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  653. [2021-03-28 05:28:36.581556] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-17: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  654. [2021-03-28 05:28:36.581693] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-11: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  655. [2021-03-28 05:28:36.582180] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-10: Connected, attached to remote volume [{conn-name=MRIData-client-10}, {remote_subvol=/gluster2/data}]
  656. [2021-03-28 05:28:36.582190] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-5: Subvolume 'MRIData-client-10' came back up; going online.
  657. [2021-03-28 05:28:36.582254] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-15: Connected, attached to remote volume [{conn-name=MRIData-client-15}, {remote_subvol=/arbiter/1}]
  658. [2021-03-28 05:28:36.582395] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-16: Connected, attached to remote volume [{conn-name=MRIData-client-16}, {remote_subvol=/arbiter/2}]
  659. [2021-03-28 05:28:36.583348] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-17: Connected, attached to remote volume [{conn-name=MRIData-client-17}, {remote_subvol=/arbiter/1}]
  660. [2021-03-28 05:28:36.583454] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-5: Client-quorum is met
  661. [2021-03-28 05:28:36.583460] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-11: Connected, attached to remote volume [{conn-name=MRIData-client-11}, {remote_subvol=/gluster3/data}]
  662. [2021-03-28 05:28:36.583983] I [fuse-bridge.c:5300:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
  663. [2021-03-28 05:28:36.584012] I [fuse-bridge.c:5928:fuse_graph_sync] 0-fuse: switched to graph 0
  664. [2021-03-28 05:28:43.797674] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x715ac) [0x7fbfa526c5ac] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x691c) [0x7fbfa4fcf91c] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7fbfa4fcf5bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for 4908ee49-69ac-4b2f-b6f8-dd2ee12180a7
  665. [2021-03-28 05:28:43.797705] W [fuse-bridge.c:1704:fuse_setattr_cbk] 0-glusterfs-fuse: 143: SETATTR() /fff => -1 (Invalid argument)
  666. [2021-03-28 05:28:52.204525] I [MSGID: 133022] [shard.c:3768:shard_delete_shards] 0-MRIData-shard: Deleted shards of gfid=4908ee49-69ac-4b2f-b6f8-dd2ee12180a7 from backend
  667. [2021-03-28 05:30:06.170864] I [fuse-bridge.c:6237:fuse_thread_proc] 0-fuse: initiating unmount of /gluster
  668. [2021-03-28 05:30:06.171009] W [glusterfsd.c:1439:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x814a) [0x7fbfacf5914a] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x55ec5d8fbb5d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x55ec5d8fb9a8] ) 0-: received signum (15), shutting down
  669. [2021-03-28 05:30:06.171029] I [fuse-bridge.c:7069:fini] 0-fuse: Unmounting '/gluster'.
  670. [2021-03-28 05:30:06.171035] I [fuse-bridge.c:7074:fini] 0-fuse: Closing fuse connection to '/gluster'.
  671. [2021-03-28 05:30:13.003659] I [MSGID: 100030] [glusterfsd.c:2689:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=8.4}, {cmdlinestr=/usr/sbin/glusterfs --acl --process-name fuse --volfile-server=hydra1 --volfile-id=/MRIData /gluster}]
  672. [2021-03-28 05:30:13.004551] I [glusterfsd.c:2424:daemonize] 0-glusterfs: Pid of current running process is 104153
  673. [2021-03-28 05:30:13.009206] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
  674. [2021-03-28 05:30:13.009625] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
  675. [2021-03-28 05:30:13.021250] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-0: parent translators are ready, attempting connect on transport []
  676. [2021-03-28 05:30:13.022697] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-1: parent translators are ready, attempting connect on transport []
  677. [2021-03-28 05:30:13.023270] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-0: changing port to 49152 (from 0)
  678. [2021-03-28 05:30:13.023286] I [socket.c:849:__socket_shutdown] 0-MRIData-client-0: intentional socket shutdown(13)
  679. [2021-03-28 05:30:13.024006] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-12: parent translators are ready, attempting connect on transport []
  680. [2021-03-28 05:30:13.024951] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-1: changing port to 49153 (from 0)
  681. [2021-03-28 05:30:13.024975] I [socket.c:849:__socket_shutdown] 0-MRIData-client-1: intentional socket shutdown(14)
  682. [2021-03-28 05:30:13.025256] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-2: parent translators are ready, attempting connect on transport []
  683. [2021-03-28 05:30:13.026275] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  684. [2021-03-28 05:30:13.026510] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-3: parent translators are ready, attempting connect on transport []
  685. [2021-03-28 05:30:13.026638] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-12: changing port to 49154 (from 0)
  686. [2021-03-28 05:30:13.026651] I [socket.c:849:__socket_shutdown] 0-MRIData-client-12: intentional socket shutdown(16)
  687. [2021-03-28 05:30:13.026664] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-1: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  688. [2021-03-28 05:30:13.027758] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-13: parent translators are ready, attempting connect on transport []
  689. [2021-03-28 05:30:13.028043] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-0: Connected, attached to remote volume [{conn-name=MRIData-client-0}, {remote_subvol=/gluster1/data}]
  690. [2021-03-28 05:30:13.028054] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-0: Subvolume 'MRIData-client-0' came back up; going online.
  691. [2021-03-28 05:30:13.028285] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-1: Connected, attached to remote volume [{conn-name=MRIData-client-1}, {remote_subvol=/gluster2/data}]
  692. [2021-03-28 05:30:13.028308] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-0: Client-quorum is met
  693. [2021-03-28 05:30:13.028509] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-2: changing port to 49155 (from 0)
  694. [2021-03-28 05:30:13.028523] I [socket.c:849:__socket_shutdown] 0-MRIData-client-2: intentional socket shutdown(14)
  695. [2021-03-28 05:30:13.029139] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-4: parent translators are ready, attempting connect on transport []
  696. [2021-03-28 05:30:13.029962] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-12: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  697. [2021-03-28 05:30:13.029999] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-3: changing port to 49152 (from 0)
  698. [2021-03-28 05:30:13.030011] I [socket.c:849:__socket_shutdown] 0-MRIData-client-3: intentional socket shutdown(17)
  699. [2021-03-28 05:30:13.030463] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-5: parent translators are ready, attempting connect on transport []
  700. [2021-03-28 05:30:13.031402] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-12: Connected, attached to remote volume [{conn-name=MRIData-client-12}, {remote_subvol=/arbiter/1}]
  701. [2021-03-28 05:30:13.031559] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-13: changing port to 49156 (from 0)
  702. [2021-03-28 05:30:13.031581] I [socket.c:849:__socket_shutdown] 0-MRIData-client-13: intentional socket shutdown(19)
  703. [2021-03-28 05:30:13.031614] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-2: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  704. [2021-03-28 05:30:13.031713] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-14: parent translators are ready, attempting connect on transport []
  705. [2021-03-28 05:30:13.032949] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-3: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  706. [2021-03-28 05:30:13.032977] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-6: parent translators are ready, attempting connect on transport []
  707. [2021-03-28 05:30:13.033026] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-4: changing port to 49153 (from 0)
  708. [2021-03-28 05:30:13.033039] I [socket.c:849:__socket_shutdown] 0-MRIData-client-4: intentional socket shutdown(20)
  709. [2021-03-28 05:30:13.033053] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-2: Connected, attached to remote volume [{conn-name=MRIData-client-2}, {remote_subvol=/gluster3/data}]
  710. [2021-03-28 05:30:13.033060] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-1: Subvolume 'MRIData-client-2' came back up; going online.
  711. [2021-03-28 05:30:13.034219] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-7: parent translators are ready, attempting connect on transport []
  712. [2021-03-28 05:30:13.034568] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-13: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  713. [2021-03-28 05:30:13.034625] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-3: Connected, attached to remote volume [{conn-name=MRIData-client-3}, {remote_subvol=/gluster1/data}]
  714. [2021-03-28 05:30:13.035442] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-15: parent translators are ready, attempting connect on transport []
  715. [2021-03-28 05:30:13.035528] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-1: Client-quorum is met
  716. [2021-03-28 05:30:13.035815] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-13: Connected, attached to remote volume [{conn-name=MRIData-client-13}, {remote_subvol=/arbiter/2}]
  717. [2021-03-28 05:30:13.035939] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-5: changing port to 49154 (from 0)
  718. [2021-03-28 05:30:13.035951] I [socket.c:849:__socket_shutdown] 0-MRIData-client-5: intentional socket shutdown(17)
  719. [2021-03-28 05:30:13.035964] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-4: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  720. [2021-03-28 05:30:13.036046] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-14: changing port to 49155 (from 0)
  721. [2021-03-28 05:30:13.036056] I [socket.c:849:__socket_shutdown] 0-MRIData-client-14: intentional socket shutdown(19)
  722. [2021-03-28 05:30:13.036697] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-8: parent translators are ready, attempting connect on transport []
  723. [2021-03-28 05:30:13.037945] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-9: parent translators are ready, attempting connect on transport []
  724. [2021-03-28 05:30:13.038691] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-4: Connected, attached to remote volume [{conn-name=MRIData-client-4}, {remote_subvol=/gluster2/data}]
  725. [2021-03-28 05:30:13.038701] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-2: Subvolume 'MRIData-client-4' came back up; going online.
  726. [2021-03-28 05:30:13.038909] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-7: changing port to 49153 (from 0)
  727. [2021-03-28 05:30:13.038947] I [socket.c:849:__socket_shutdown] 0-MRIData-client-7: intentional socket shutdown(20)
  728. [2021-03-28 05:30:13.038958] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-6: changing port to 49152 (from 0)
  729. [2021-03-28 05:30:13.038968] I [socket.c:849:__socket_shutdown] 0-MRIData-client-6: intentional socket shutdown(21)
  730. [2021-03-28 05:30:13.039186] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-16: parent translators are ready, attempting connect on transport []
  731. [2021-03-28 05:30:13.040453] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-10: parent translators are ready, attempting connect on transport []
  732. [2021-03-28 05:30:13.041617] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-5: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  733. [2021-03-28 05:30:13.041683] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-14: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  734. [2021-03-28 05:30:13.041684] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-11: parent translators are ready, attempting connect on transport []
  735. [2021-03-28 05:30:13.041789] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-15: changing port to 49155 (from 0)
  736. [2021-03-28 05:30:13.041804] I [socket.c:849:__socket_shutdown] 0-MRIData-client-15: intentional socket shutdown(24)
  737. [2021-03-28 05:30:13.041846] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-8: changing port to 49154 (from 0)
  738. [2021-03-28 05:30:13.041856] I [socket.c:849:__socket_shutdown] 0-MRIData-client-8: intentional socket shutdown(26)
  739. [2021-03-28 05:30:13.042910] I [MSGID: 114020] [client.c:2315:notify] 0-MRIData-client-17: parent translators are ready, attempting connect on transport []
  740. [2021-03-28 05:30:13.043164] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-6: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  741. Final graph:
  742. +------------------------------------------------------------------------------+
  743. 1: volume MRIData-client-0
  744. 2: type protocol/client
  745. 3: option opversion 80000
  746. 4: option clnt-lk-version 1
  747. 5: option volfile-checksum 0
  748. 6: option volfile-key /MRIData
  749. 7: option client-version 8.4
  750. 8: option process-name fuse
  751. 9: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-0-RECON_NO:-0
  752. 10: option fops-version 1298437
  753. 11: option ping-timeout 10
  754. 12: option remote-host hydra1
  755. 13: option remote-subvolume /gluster1/data
  756. 14: option transport-type socket
  757. 15: option transport.address-family inet
  758. 16: option filter-O_DIRECT enable
  759. 17: option transport.tcp-user-timeout 0
  760. 18: option transport.socket.keepalive-time 20
  761. 19: option transport.socket.keepalive-interval 2
  762. 20: option transport.socket.keepalive-count 9
  763. 21: option send-gids true
  764. 22: end-volume
  765. 23:
  766. 24: volume MRIData-client-1
  767. 25: type protocol/client
  768. 26: option opversion 80000
  769. 27: option clnt-lk-version 1
  770. 28: option volfile-checksum 0
  771. 29: option volfile-key /MRIData
  772. 30: option client-version 8.4
  773. 31: option process-name fuse
  774. 32: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-1-RECON_NO:-0
  775. 33: option fops-version 1298437
  776. 34: option ping-timeout 10
  777. 35: option remote-host hydra1
  778. 36: option remote-subvolume /gluster2/data
  779. 37: option transport-type socket
  780. 38: option transport.address-family inet
  781. 39: option filter-O_DIRECT enable
  782. 40: option transport.tcp-user-timeout 0
  783. 41: option transport.socket.keepalive-time 20
  784. 42: option transport.socket.keepalive-interval 2
  785. 43: option transport.socket.keepalive-count 9
  786. 44: option send-gids true
  787. 45: end-volume
  788. 46:
  789. 47: volume MRIData-client-12
  790. 48: type protocol/client
  791. 49: option opversion 80000
  792. 50: option clnt-lk-version 1
  793. 51: option volfile-checksum 0
  794. 52: option volfile-key /MRIData
  795. 53: option client-version 8.4
  796. 54: option process-name fuse
  797. 55: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-12-RECON_NO:-0
  798. 56: option fops-version 1298437
  799. 57: option ping-timeout 10
  800. 58: option remote-host hydra1
  801. 59: option remote-subvolume /arbiter/1
  802. 60: option transport-type socket
  803. 61: option transport.address-family inet
  804. 62: option filter-O_DIRECT enable
  805. 63: option transport.tcp-user-timeout 0
  806. 64: option transport.socket.keepalive-time 20
  807. 65: option transport.socket.keepalive-interval 2
  808. 66: option transport.socket.keepalive-count 9
  809. 67: option send-gids true
  810. 68: end-volume
  811. 69:
  812. 70: volume MRIData-replicate-0
  813. 71: type cluster/replicate
  814. 72: option afr-pending-xattr MRIData-client-0,MRIData-client-1,MRIData-client-12
  815. 73: option arbiter-count 1
  816. 74: option metadata-self-heal on
  817. 75: option data-self-heal on
  818. 76: option entry-self-heal on
  819. 77: option data-self-heal-algorithm full
  820. 78: option eager-lock enable
  821. 79: option quorum-type auto
  822. 80: option choose-local off
  823. 81: option shd-max-threads 8
  824. 82: option shd-wait-qlength 10000
  825. 83: option locking-scheme granular
  826. 84: option use-compound-fops off
  827. 85: subvolumes MRIData-client-0 MRIData-client-1 MRIData-client-12
  828. 86: end-volume
  829. 87:
  830. 88: volume MRIData-client-2
  831. 89: type protocol/client
  832. 90: option opversion 80000
  833. 91: option clnt-lk-version 1
  834. 92: option volfile-checksum 0
  835. 93: option volfile-key /MRIData
  836. 94: option client-version 8.4
  837. 95: option process-name fuse
  838. 96: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-2-RECON_NO:-0
  839. 97: option fops-version 1298437
  840. 98: option ping-timeout 10
  841. 99: option remote-host hydra1
  842. 100: option remote-subvolume /gluster3/data
  843. 101: option transport-type socket
  844. 102: option transport.address-family inet
  845. 103: option filter-O_DIRECT enable
  846. 104: option transport.tcp-user-timeout 0
  847. 105: option transport.socket.keepalive-time 20
  848. 106: option transport.socket.keepalive-interval 2
  849. 107: option transport.socket.keepalive-count 9
  850. 108: option send-gids true
  851. 109: end-volume
  852. 110:
  853. 111: volume MRIData-client-3
  854. 112: type protocol/client
  855. 113: option opversion 80000
  856. 114: option clnt-lk-version 1
  857. 115: option volfile-checksum 0
  858. 116: option volfile-key /MRIData
  859. 117: option client-version 8.4
  860. 118: option process-name fuse
  861. 119: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-3-RECON_NO:-0
  862. 120: option fops-version 1298437
  863. 121: option ping-timeout 10
  864. 122: option remote-host hydra2
  865. 123: option remote-subvolume /gluster1/data
  866. 124: option transport-type socket
  867. 125: option transport.address-family inet
  868. 126: option filter-O_DIRECT enable
  869. 127: option transport.tcp-user-timeout 0
  870. 128: option transport.socket.keepalive-time 20
  871. 129: option transport.socket.keepalive-interval 2
  872. 130: option transport.socket.keepalive-count 9
  873. 131: option send-gids true
  874. 132: end-volume
  875. 133:
  876. 134: volume MRIData-client-13
  877. 135: type protocol/client
  878. 136: option opversion 80000
  879. 137: option clnt-lk-version 1
  880. 138: option volfile-checksum 0
  881. 139: option volfile-key /MRIData
  882. 140: option client-version 8.4
  883. 141: option process-name fuse
  884. 142: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-13-RECON_NO:-0
  885. 143: option fops-version 1298437
  886. 144: option ping-timeout 10
  887. 145: option remote-host hydra1
  888. 146: option remote-subvolume /arbiter/2
  889. 147: option transport-type socket
  890. 148: option transport.address-family inet
  891. 149: option filter-O_DIRECT enable
  892. 150: option transport.tcp-user-timeout 0
  893. 151: option transport.socket.keepalive-time 20
  894. 152: option transport.socket.keepalive-interval 2
  895. 153: option transport.socket.keepalive-count 9
  896. 154: option send-gids true
  897. 155: end-volume
  898. 156:
  899. 157: volume MRIData-replicate-1
  900. 158: type cluster/replicate
  901. 159: option afr-pending-xattr MRIData-client-2,MRIData-client-3,MRIData-client-13
  902. 160: option arbiter-count 1
  903. 161: option metadata-self-heal on
  904. 162: option data-self-heal on
  905. 163: option entry-self-heal on
  906. 164: option data-self-heal-algorithm full
  907. 165: option eager-lock enable
  908. 166: option quorum-type auto
  909. 167: option choose-local off
  910. 168: option shd-max-threads 8
  911. 169: option shd-wait-qlength 10000
  912. 170: option locking-scheme granular
  913. 171: option use-compound-fops off
  914. 172: subvolumes MRIData-client-2 MRIData-client-3 MRIData-client-13
  915. 173: end-volume
  916. 174:
  917. 175: volume MRIData-client-4
  918. 176: type protocol/client
  919. 177: option opversion 80000
  920. 178: option clnt-lk-version 1
  921. 179: option volfile-checksum 0
  922. 180: option volfile-key /MRIData
  923. 181: option client-version 8.4
  924. 182: option process-name fuse
  925. 183: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-4-RECON_NO:-0
  926. 184: option fops-version 1298437
  927. 185: option ping-timeout 10
  928. 186: option remote-host hydra2
  929. 187: option remote-subvolume /gluster2/data
  930. 188: option transport-type socket
  931. 189: option transport.address-family inet
  932. 190: option filter-O_DIRECT enable
  933. 191: option transport.tcp-user-timeout 0
  934. 192: option transport.socket.keepalive-time 20
  935. 193: option transport.socket.keepalive-interval 2
  936. 194: option transport.socket.keepalive-count 9
  937. 195: option send-gids true
  938. 196: end-volume
  939. 197:
  940. 198: volume MRIData-client-5
  941. 199: type protocol/client
  942. 200: option opversion 80000
  943. 201: option clnt-lk-version 1
  944. 202: option volfile-checksum 0
  945. 203: option volfile-key /MRIData
  946. 204: option client-version 8.4
  947. 205: option process-name fuse
  948. 206: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-5-RECON_NO:-0
  949. 207: option fops-version 1298437
  950. 208: option ping-timeout 10
  951. 209: option remote-host hydra2
  952. 210: option remote-subvolume /gluster3/data
  953. 211: option transport-type socket
  954. 212: option transport.address-family inet
  955. 213: option filter-O_DIRECT enable
  956. 214: option transport.tcp-user-timeout 0
  957. 215: option transport.socket.keepalive-time 20
  958. 216: option transport.socket.keepalive-interval 2
  959. 217: option transport.socket.keepalive-count 9
  960. 218: option send-gids true
  961. 219: end-volume
  962. 220:
  963. 221: volume MRIData-client-14
  964. 222: type protocol/client
  965. 223: option opversion 80000
  966. 224: option clnt-lk-version 1
  967. 225: option volfile-checksum 0
  968. 226: option volfile-key /MRIData
  969. 227: option client-version 8.4
  970. 228: option process-name fuse
  971. 229: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-14-RECON_NO:-0
  972. 230: option fops-version 1298437
  973. 231: option ping-timeout 10
  974. 232: option remote-host hydra2
  975. 233: option remote-subvolume /arbiter/1
  976. 234: option transport-type socket
  977. 235: option transport.address-family inet
  978. 236: option filter-O_DIRECT enable
  979. 237: option transport.tcp-user-timeout 0
  980. 238: option transport.socket.keepalive-time 20
  981. 239: option transport.socket.keepalive-interval 2
  982. 240: option transport.socket.keepalive-count 9
  983. 241: option send-gids true
  984. 242: end-volume
  985. 243:
  986. 244: volume MRIData-replicate-2
  987. 245: type cluster/replicate
  988. 246: option afr-pending-xattr MRIData-client-4,MRIData-client-5,MRIData-client-14
  989. 247: option arbiter-count 1
  990. 248: option metadata-self-heal on
  991. 249: option data-self-heal on
  992. 250: option entry-self-heal on
  993. 251: option data-self-heal-algorithm full
  994. 252: option eager-lock enable
  995. 253: option quorum-type auto
  996. 254: option choose-local off
  997. 255: option shd-max-threads 8
  998. 256: option shd-wait-qlength 10000
  999. 257: option locking-scheme granular
  1000. 258: option use-compound-fops off
  1001. 259: subvolumes MRIData-client-4 MRIData-client-5 MRIData-client-14
  1002. 260: end-volume
  1003. 261:
  1004. 262: volume MRIData-client-6
  1005. 263: type protocol/client
  1006. 264: option opversion 80000
  1007. 265: option clnt-lk-version 1
  1008. 266: option volfile-checksum 0
  1009. 267: option volfile-key /MRIData
  1010. 268: option client-version 8.4
  1011. 269: option process-name fuse
  1012. 270: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-6-RECON_NO:-0
  1013. 271: option fops-version 1298437
  1014. 272: option ping-timeout 10
  1015. 273: option remote-host hydra3
  1016. 274: option remote-subvolume /gluster1/data
  1017. [2021-03-28 05:30:13.044457] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-7: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1018. 275: option transport-type socket
  1019. 276: option transport.address-family inet
  1020. 277: option filter-O_DIRECT enable
  1021. 278: option transport.tcp-user-timeout 0
  1022. 279: option transport.socket.keepalive-time 20
  1023. 280: option transport.socket.keepalive-interval 2
  1024. 281: option transport.socket.keepalive-count 9
  1025. 282: option send-gids true
  1026. 283: end-volume
  1027. 284:
  1028. 285: volume MRIData-client-7
  1029. 286: type protocol/client
  1030. 287: option opversion 80000
  1031. 288: option clnt-lk-version 1
  1032. 289: option volfile-checksum 0
  1033. 290: option volfile-key /MRIData
  1034. 291: option client-version 8.4
  1035. 292: option process-name fuse
  1036. 293: option process-uuid CTX_ID:a9b76205-4dd0-41db-beb7-ed0f7eaf2c08-GRAPH_ID:0-PID:104153-HOST:argus.cbi.fas.nyu.edu-PC_NAME:MRIData-client-7-RECON_NO:-0
  1037. 294: option fops-version 1298437
  1038. 295: option ping-timeout 10
  1039. 296: option remote-host hydra3
  1040. 297: option remote-subvolume /gluster2/data
  1041. 298: option transport-type socket
  1042. 299: option transport.address-family inet
  1043. 300: option filter-O_DIRECT enable
  1044. 301: option transport.tcp-user-timeout 0
  1045. 302: option transport.socket.keepalive-time 20
  1046. 303: option transport.socket.keepalive-interval 2
  1047. 304: option transport.socket.keepalive-count 9
  1048. 305: option send-gids true
  1049. 306: end-volume
  1050. 307:
  1051. 308: volume MRIData-client-15
  1052. 309: type protocol/client
  1053. 310: option ping-timeout 10
  1054. 311: option remote-host hydra3
  1055. 312: option remote-subvolume /arbiter/1
  1056. 313: option transport-type socket
  1057. 314: option transport.address-family inet
  1058. 315: option filter-O_DIRECT enable
  1059. 316: option transport.tcp-user-timeout 0
  1060. 317: option transport.socket.keepalive-time 20
  1061. 318: option transport.socket.keepalive-interval 2
  1062. 319: option transport.socket.keepalive-count 9
  1063. 320: option send-gids true
  1064. 321: end-volume
  1065. 322:
  1066. 323: volume MRIData-replicate-3
  1067. 324: type cluster/replicate
  1068. 325: option afr-pending-xattr MRIData-client-6,MRIData-client-7,MRIData-client-15
  1069. 326: option arbiter-count 1
  1070. 327: option metadata-self-heal on
  1071. 328: option data-self-heal on
  1072. 329: option entry-self-heal on
  1073. 330: option data-self-heal-algorithm full
  1074. 331: option eager-lock enable
  1075. 332: option quorum-type auto
  1076. 333: option choose-local off
  1077. 334: option shd-max-threads 8
  1078. 335: option shd-wait-qlength 10000
  1079. 336: option locking-scheme granular
  1080. 337: option use-compound-fops off
  1081. [2021-03-28 05:30:13.044542] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-5: Connected, attached to remote volume [{conn-name=MRIData-client-5}, {remote_subvol=/gluster3/data}]
  1082. 338: subvolumes MRIData-client-6 MRIData-client-7 MRIData-client-15
  1083. 339: end-volume
  1084. 340:
  1085. 341: volume MRIData-client-8
  1086. 342: type protocol/client
  1087. 343: option ping-timeout 10
  1088. 344: option remote-host hydra3
  1089. 345: option remote-subvolume /gluster3/data
  1090. 346: option transport-type socket
  1091. 347: option transport.address-family inet
  1092. 348: option filter-O_DIRECT enable
  1093. 349: option transport.tcp-user-timeout 0
  1094. 350: option transport.socket.keepalive-time 20
  1095. 351: option transport.socket.keepalive-interval 2
  1096. 352: option transport.socket.keepalive-count 9
  1097. 353: option send-gids true
  1098. 354: end-volume
  1099. 355:
  1100. 356: volume MRIData-client-9
  1101. 357: type protocol/client
  1102. 358: option ping-timeout 10
  1103. 359: option remote-host hydra4
  1104. 360: option remote-subvolume /gluster1/data
  1105. 361: option transport-type socket
  1106. 362: option transport.address-family inet
  1107. 363: option filter-O_DIRECT enable
  1108. 364: option transport.tcp-user-timeout 0
  1109. 365: option transport.socket.keepalive-time 20
  1110. 366: option transport.socket.keepalive-interval 2
  1111. 367: option transport.socket.keepalive-count 9
  1112. 368: option send-gids true
  1113. 369: end-volume
  1114. 370:
  1115. 371: volume MRIData-client-16
  1116. 372: type protocol/client
  1117. 373: option ping-timeout 10
  1118. 374: option remote-host hydra3
  1119. 375: option remote-subvolume /arbiter/2
  1120. 376: option transport-type socket
  1121. 377: option transport.address-family inet
  1122. 378: option filter-O_DIRECT enable
  1123. 379: option transport.tcp-user-timeout 0
  1124. 380: option transport.socket.keepalive-time 20
  1125. 381: option transport.socket.keepalive-interval 2
  1126. 382: option transport.socket.keepalive-count 9
  1127. 383: option send-gids true
  1128. 384: end-volume
  1129. 385:
  1130. 386: volume MRIData-replicate-4
  1131. 387: type cluster/replicate
  1132. 388: option afr-pending-xattr MRIData-client-8,MRIData-client-9,MRIData-client-16
  1133. 389: option arbiter-count 1
  1134. 390: option metadata-self-heal on
  1135. 391: option data-self-heal on
  1136. 392: option entry-self-heal on
  1137. 393: option data-self-heal-algorithm full
  1138. 394: option eager-lock enable
  1139. 395: option quorum-type auto
  1140. 396: option choose-local off
  1141. 397: option shd-max-threads 8
  1142. 398: option shd-wait-qlength 10000
  1143. 399: option locking-scheme granular
  1144. 400: option use-compound-fops off
  1145. 401: subvolumes MRIData-client-8 MRIData-client-9 MRIData-client-16
  1146. 402: end-volume
  1147. 403:
  1148. 404: volume MRIData-client-10
  1149. 405: type protocol/client
  1150. 406: option ping-timeout 10
  1151. 407: option remote-host hydra4
  1152. 408: option remote-subvolume /gluster2/data
  1153. 409: option transport-type socket
  1154. 410: option transport.address-family inet
  1155. 411: option filter-O_DIRECT enable
  1156. 412: option transport.tcp-user-timeout 0
  1157. 413: option transport.socket.keepalive-time 20
  1158. 414: option transport.socket.keepalive-interval 2
  1159. 415: option transport.socket.keepalive-count 9
  1160. 416: option send-gids true
  1161. 417: end-volume
  1162. 418:
  1163. 419: volume MRIData-client-11
  1164. 420: type protocol/client
  1165. 421: option ping-timeout 10
  1166. 422: option remote-host hydra4
  1167. [2021-03-28 05:30:13.044649] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-2: Client-quorum is met
  1168. 423: option remote-subvolume /gluster3/data
  1169. 424: option transport-type socket
  1170. 425: option transport.address-family inet
  1171. 426: option filter-O_DIRECT enable
  1172. 427: option transport.tcp-user-timeout 0
  1173. 428: option transport.socket.keepalive-time 20
  1174. 429: option transport.socket.keepalive-interval 2
  1175. 430: option transport.socket.keepalive-count 9
  1176. 431: option send-gids true
  1177. 432: end-volume
  1178. 433:
  1179. 434: volume MRIData-client-17
  1180. 435: type protocol/client
  1181. 436: option ping-timeout 10
  1182. 437: option remote-host hydra4
  1183. 438: option remote-subvolume /arbiter/1
  1184. 439: option transport-type socket
  1185. 440: option transport.address-family inet
  1186. 441: option filter-O_DIRECT enable
  1187. 442: option transport.tcp-user-timeout 0
  1188. 443: option transport.socket.keepalive-time 20
  1189. 444: option transport.socket.keepalive-interval 2
  1190. 445: option transport.socket.keepalive-count 9
  1191. 446: option send-gids true
  1192. 447: end-volume
  1193. 448:
  1194. 449: volume MRIData-replicate-5
  1195. 450: type cluster/replicate
  1196. 451: option afr-pending-xattr MRIData-client-10,MRIData-client-11,MRIData-client-17
  1197. 452: option arbiter-count 1
  1198. 453: option metadata-self-heal on
  1199. 454: option data-self-heal on
  1200. 455: option entry-self-heal on
  1201. 456: option data-self-heal-algorithm full
  1202. 457: option eager-lock enable
  1203. 458: option quorum-type auto
  1204. 459: option choose-local off
  1205. 460: option shd-max-threads 8
  1206. 461: option shd-wait-qlength 10000
  1207. 462: option locking-scheme granular
  1208. 463: option use-compound-fops off
  1209. 464: subvolumes MRIData-client-10 MRIData-client-11 MRIData-client-17
  1210. 465: end-volume
  1211. 466:
  1212. 467: volume MRIData-dht
  1213. 468: type cluster/distribute
  1214. 469: option lock-migration off
  1215. 470: option force-migration off
  1216. 471: subvolumes MRIData-replicate-0 MRIData-replicate-1 MRIData-replicate-2 MRIData-replicate-3 MRIData-replicate-4 MRIData-replicate-5
  1217. 472: end-volume
  1218. 473:
  1219. 474: volume MRIData-shard
  1220. 475: type features/shard
  1221. 476: subvolumes MRIData-dht
  1222. 477: end-volume
  1223. 478:
  1224. 479: volume MRIData-write-behind
  1225. 480: type performance/write-behind
  1226. 481: subvolumes MRIData-shard
  1227. 482: end-volume
  1228. 483:
  1229. 484: volume MRIData-readdir-ahead
  1230. 485: type performance/readdir-ahead
  1231. 486: option parallel-readdir off
  1232. 487: option rda-request-size 131072
  1233. 488: option rda-cache-limit 10MB
  1234. 489: subvolumes MRIData-write-behind
  1235. 490: end-volume
  1236. 491:
  1237. 492: volume MRIData-open-behind
  1238. 493: type performance/open-behind
  1239. 494: subvolumes MRIData-readdir-ahead
  1240. 495: end-volume
  1241. 496:
  1242. 497: volume MRIData-md-cache
  1243. 498: type performance/md-cache
  1244. 499: option cache-posix-acl true
  1245. 500: subvolumes MRIData-open-behind
  1246. 501: end-volume
  1247. 502:
  1248. 503: volume MRIData
  1249. 504: type debug/io-stats
  1250. 505: option log-level INFO
  1251. 506: option latency-measurement off
  1252. 507: option count-fop-hits off
  1253. 508: subvolumes MRIData-md-cache
  1254. 509: end-volume
  1255. 510:
  1256. 511: volume posix-acl-autoload
  1257. 512: type system/posix-acl
  1258. 513: subvolumes MRIData
  1259. 514: end-volume
  1260. 515:
  1261. 516: volume meta-autoload
  1262. 517: type meta
  1263. 518: subvolumes posix-acl-autoload
  1264. 519: end-volume
  1265. 520:
  1266. +------------------------------------------------------------------------------+
  1267. [2021-03-28 05:30:13.044855] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-14: Connected, attached to remote volume [{conn-name=MRIData-client-14}, {remote_subvol=/arbiter/1}]
  1268. [2021-03-28 05:30:13.044979] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-6: Connected, attached to remote volume [{conn-name=MRIData-client-6}, {remote_subvol=/gluster1/data}]
  1269. [2021-03-28 05:30:13.044987] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-9: changing port to 49152 (from 0)
  1270. [2021-03-28 05:30:13.044989] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-3: Subvolume 'MRIData-client-6' came back up; going online.
  1271. [2021-03-28 05:30:13.045000] I [socket.c:849:__socket_shutdown] 0-MRIData-client-9: intentional socket shutdown(27)
  1272. [2021-03-28 05:30:13.045009] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-16: changing port to 49156 (from 0)
  1273. [2021-03-28 05:30:13.045018] I [socket.c:849:__socket_shutdown] 0-MRIData-client-16: intentional socket shutdown(28)
  1274. [2021-03-28 05:30:13.045134] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-10: changing port to 49153 (from 0)
  1275. [2021-03-28 05:30:13.045148] I [socket.c:849:__socket_shutdown] 0-MRIData-client-10: intentional socket shutdown(21)
  1276. [2021-03-28 05:30:13.046328] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-15: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1277. [2021-03-28 05:30:13.046387] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-8: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1278. [2021-03-28 05:30:13.046500] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-7: Connected, attached to remote volume [{conn-name=MRIData-client-7}, {remote_subvol=/gluster2/data}]
  1279. [2021-03-28 05:30:13.047448] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-3: Client-quorum is met
  1280. [2021-03-28 05:30:13.047692] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-11: changing port to 49154 (from 0)
  1281. [2021-03-28 05:30:13.047694] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-15: Connected, attached to remote volume [{conn-name=MRIData-client-15}, {remote_subvol=/arbiter/1}]
  1282. [2021-03-28 05:30:13.047706] I [socket.c:849:__socket_shutdown] 0-MRIData-client-11: intentional socket shutdown(29)
  1283. [2021-03-28 05:30:13.047722] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-8: Connected, attached to remote volume [{conn-name=MRIData-client-8}, {remote_subvol=/gluster3/data}]
  1284. [2021-03-28 05:30:13.047839] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-4: Subvolume 'MRIData-client-8' came back up; going online.
  1285. [2021-03-28 05:30:13.047855] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-9: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1286. [2021-03-28 05:30:13.047886] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 0-MRIData-client-17: changing port to 49155 (from 0)
  1287. [2021-03-28 05:30:13.047898] I [socket.c:849:__socket_shutdown] 0-MRIData-client-17: intentional socket shutdown(31)
  1288. [2021-03-28 05:30:13.047911] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-16: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1289. [2021-03-28 05:30:13.049245] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-10: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1290. [2021-03-28 05:30:13.049308] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-9: Connected, attached to remote volume [{conn-name=MRIData-client-9}, {remote_subvol=/gluster1/data}]
  1291. [2021-03-28 05:30:13.049395] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-16: Connected, attached to remote volume [{conn-name=MRIData-client-16}, {remote_subvol=/arbiter/2}]
  1292. [2021-03-28 05:30:13.049409] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-4: Client-quorum is met
  1293. [2021-03-28 05:30:13.049718] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-11: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1294. [2021-03-28 05:30:13.049739] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-MRIData-client-17: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
  1295. [2021-03-28 05:30:13.049977] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-10: Connected, attached to remote volume [{conn-name=MRIData-client-10}, {remote_subvol=/gluster2/data}]
  1296. [2021-03-28 05:30:13.049984] I [MSGID: 108005] [afr-common.c:5996:__afr_handle_child_up_event] 0-MRIData-replicate-5: Subvolume 'MRIData-client-10' came back up; going online.
  1297. [2021-03-28 05:30:13.050449] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-11: Connected, attached to remote volume [{conn-name=MRIData-client-11}, {remote_subvol=/gluster3/data}]
  1298. [2021-03-28 05:30:13.051373] I [MSGID: 108002] [afr-common.c:6366:afr_notify] 0-MRIData-replicate-5: Client-quorum is met
  1299. [2021-03-28 05:30:13.052240] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-MRIData-client-17: Connected, attached to remote volume [{conn-name=MRIData-client-17}, {remote_subvol=/arbiter/1}]
  1300. [2021-03-28 05:30:13.052719] I [fuse-bridge.c:5300:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
  1301. [2021-03-28 05:30:13.052729] I [fuse-bridge.c:5928:fuse_graph_sync] 0-fuse: switched to graph 0
  1302. [2021-03-28 05:30:17.671109] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x715ac) [0x7f47817b55ac] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x691c) [0x7f478151891c] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1303. [2021-03-28 05:30:17.671133] W [fuse-bridge.c:1704:fuse_setattr_cbk] 0-glusterfs-fuse: 141: SETATTR() /fff => -1 (Invalid argument)
  1304. [2021-03-28 05:31:07.093833] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x715ac) [0x7f47817b55ac] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x691c) [0x7f478151891c] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1305. [2021-03-28 05:31:07.093879] W [fuse-bridge.c:1704:fuse_setattr_cbk] 0-glusterfs-fuse: 153: SETATTR() /fff => -1 (Invalid argument)
  1306. [2021-03-28 05:43:07.978142] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f47817c2b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f4781518767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1307. [2021-03-28 05:43:07.978172] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 204: STAT() /fff => -1 (Invalid argument)
  1308. [2021-03-28 05:43:12.471539] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f47817c2b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f4781518767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1309. [2021-03-28 05:43:12.471563] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 257: STAT() /fff => -1 (Invalid argument)
  1310. [2021-03-28 05:43:12.485238] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f47817c2b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f4781518767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1311. [2021-03-28 05:43:12.485264] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 281: STAT() /fff => -1 (Invalid argument)
  1312. [2021-03-28 05:43:12.485821] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f47817c2b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f4781518767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1313. [2021-03-28 05:43:12.485832] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 282: STAT() /fff => -1 (Invalid argument)
  1314. [2021-03-28 05:43:13.194276] E [shard.c:563:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/8.4/xlator/cluster/distribute.so(+0x7eb84) [0x7f47817c2b84] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x6767) [0x7f4781518767] -->/usr/lib64/glusterfs/8.4/xlator/features/shard.so(+0x65bd) [0x7f47815185bd] ) 0-MRIData-shard: Failed to get trusted.glusterfs.shard.file-size for f83d1e25-9431-4258-ab0f-45d79c4d187a
  1315. [2021-03-28 05:43:13.194339] W [fuse-bridge.c:1387:fuse_attr_cbk] 0-glusterfs-fuse: 283: STAT() /fff => -1 (Invalid argument)
  1316. [2021-03-28 05:43:13.230287] I [MSGID: 133022] [shard.c:3768:shard_delete_shards] 0-MRIData-shard: Deleted shards of gfid=f83d1e25-9431-4258-ab0f-45d79c4d187a from backend
  1317. [2021-03-28 07:39:02.567466] I [MSGID: 100011] [glusterfsd.c:1521:reincarnate] 0-glusterfsd: Fetching the volume file from server... []
RAW Paste Data