[2015-11-24 09:43:23.396157] E [socket.c:2278:socket_connect_finish] 0-export-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.398830] E [socket.c:2278:socket_connect_finish] 0-export-client-2: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.401618] E [socket.c:2278:socket_connect_finish] 0-meta-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.404320] E [socket.c:2278:socket_connect_finish] 0-data-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.407068] E [socket.c:2278:socket_connect_finish] 0-storage-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.409808] E [socket.c:2278:socket_connect_finish] 0-data-client-2: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.412624] E [socket.c:2278:socket_connect_finish] 0-storage-client-2: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.415335] E [socket.c:2278:socket_connect_finish] 0-iso-client-2: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.418134] E [socket.c:2278:socket_connect_finish] 0-iso-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.420842] E [socket.c:2278:socket_connect_finish] 0-testvol-client-0: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.423582] E [socket.c:2278:socket_connect_finish] 0-data-client-4: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 09:43:23.426384] E [socket.c:2278:socket_connect_finish] 0-testvol-client-2: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 10:02:58.695522] W [socket.c:588:__socket_rwv] 0-data-client-5: readv on 139.64.206.101:50203 failed (No data available)
[2015-11-24 10:02:58.695578] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-5: disconnected from data-client-5. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.695606] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-data-replicate-1: Client-quorum is not met
[2015-11-24 10:02:58.695874] W [socket.c:588:__socket_rwv] 0-data-client-3: readv on 139.64.206.101:50198 failed (No data available)
[2015-11-24 10:02:58.695903] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-3: disconnected from data-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.695919] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-data-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.696092] W [socket.c:588:__socket_rwv] 0-data-client-1: readv on 139.64.206.101:50190 failed (No data available)
[2015-11-24 10:02:58.696119] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-1: disconnected from data-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.696132] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-data-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.699349] W [socket.c:588:__socket_rwv] 0-export-client-1: readv on 139.64.206.101:50194 failed (No data available)
[2015-11-24 10:02:58.699411] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-1: disconnected from export-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.699432] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-export-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.702968] W [socket.c:588:__socket_rwv] 0-iso-client-1: readv on 139.64.206.101:50196 failed (No data available)
[2015-11-24 10:02:58.703002] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-1: disconnected from iso-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.703019] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-iso-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.703581] W [socket.c:588:__socket_rwv] 0-meta-client-1: readv on 139.64.206.101:50200 failed (No data available)
[2015-11-24 10:02:58.703611] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-meta-client-1: disconnected from meta-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.703627] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-meta-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.704415] W [socket.c:588:__socket_rwv] 0-export-client-3: readv on 139.64.206.101:50195 failed (No data available)
[2015-11-24 10:02:58.704460] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-3: disconnected from export-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.704477] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-export-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.705029] W [socket.c:588:__socket_rwv] 0-storage-client-1: readv on 139.64.206.101:50186 failed (No data available)
[2015-11-24 10:02:58.705029] W [socket.c:588:__socket_rwv] 0-iso-client-3: readv on 139.64.206.101:50197 failed (No data available)
[2015-11-24 10:02:58.705061] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-1: disconnected from storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.705070] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-3: disconnected from iso-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.705078] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-storage-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.705098] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-iso-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.705884] W [socket.c:588:__socket_rwv] 0-storage-client-3: readv on 139.64.206.101:50187 failed (No data available)
[2015-11-24 10:02:58.705920] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-3: disconnected from storage-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.705941] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-storage-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.708448] W [socket.c:588:__socket_rwv] 0-testvol-client-3: readv on 139.64.206.101:50189 failed (No data available)
[2015-11-24 10:02:58.708479] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-3: disconnected from testvol-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.708497] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-testvol-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:02:58.709289] W [socket.c:588:__socket_rwv] 0-testvol-client-1: readv on 139.64.206.101:50188 failed (No data available)
[2015-11-24 10:02:58.709334] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-1: disconnected from testvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:02:58.709350] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-testvol-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:03:03.532268] W [socket.c:588:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
[2015-11-24 10:03:09.210874] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-meta-client-0: changing port to 49162 (from 0)
[2015-11-24 10:03:09.210895] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-2: changing port to 49159 (from 0)
[2015-11-24 10:03:09.210954] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-0: changing port to 49158 (from 0)
[2015-11-24 10:03:09.211217] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)
[2015-11-24 10:03:09.211253] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-2: changing port to 49154 (from 0)
[2015-11-24 10:03:09.211238] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-0: changing port to 49163 (from 0)
[2015-11-24 10:03:09.211446] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-2: changing port to 49164 (from 0)
[2015-11-24 10:03:09.211483] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-2: changing port to 49161 (from 0)
[2015-11-24 10:03:09.211521] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-0: changing port to 49160 (from 0)
[2015-11-24 10:03:09.214384] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49165 (from 0)
[2015-11-24 10:03:09.216536] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-4: changing port to 49156 (from 0)
[2015-11-24 10:03:09.216568] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-2: changing port to 49166 (from 0)
[2015-11-24 10:03:09.223996] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.224211] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.224278] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.224336] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.224383] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.224693] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-0: Connected to export-client-0, attached to remote volume '/tank/export/brick1'.
[2015-11-24 10:03:09.224709] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.224754] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-export-replicate-0: Subvolume 'export-client-0' came back up; going online.
[2015-11-24 10:03:09.224764] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-export-replicate-0: Client-quorum is met
[2015-11-24 10:03:09.224825] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-0: Connected to storage-client-0, attached to remote volume '/tank/storage/brick1'.
[2015-11-24 10:03:09.224835] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.224874] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-storage-replicate-0: Subvolume 'storage-client-0' came back up; going online.
[2015-11-24 10:03:09.224884] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-storage-replicate-0: Client-quorum is met
[2015-11-24 10:03:09.225069] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-0: Server lk version = 1
[2015-11-24 10:03:09.225117] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-0: Server lk version = 1
[2015-11-24 10:03:09.227535] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-meta-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.228238] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-2: Connected to export-client-2, attached to remote volume '/tank/export/brick2'.
[2015-11-24 10:03:09.228265] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.228289] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-2: Connected to data-client-2, attached to remote volume '/tank/data/brick2'.
[2015-11-24 10:03:09.228313] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.228322] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-export-replicate-1: Subvolume 'export-client-2' came back up; going online.
[2015-11-24 10:03:09.228337] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-export-replicate-1: Client-quorum is met
[2015-11-24 10:03:09.228401] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/tank/data/brick1'.
[2015-11-24 10:03:09.228412] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.228427] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-data-replicate-0: Subvolume 'data-client-2' came back up; going online.
[2015-11-24 10:03:09.228544] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-data-replicate-0: Client-quorum is met
[2015-11-24 10:03:09.228589] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-2: Server lk version = 1
[2015-11-24 10:03:09.228619] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-2: Server lk version = 1
[2015-11-24 10:03:09.228941] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-0: Server lk version = 1
[2015-11-24 10:03:09.230479] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.231390] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-meta-client-0: Connected to meta-client-0, attached to remote volume '/tank/meta/brick1'.
[2015-11-24 10:03:09.231409] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-meta-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.231447] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-meta-replicate-0: Subvolume 'meta-client-0' came back up; going online.
[2015-11-24 10:03:09.231688] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-meta-client-0: Server lk version = 1
[2015-11-24 10:03:09.232807] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.233229] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-2: Connected to iso-client-2, attached to remote volume '/tank/iso/brick2'.
[2015-11-24 10:03:09.233253] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.233291] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-iso-replicate-1: Subvolume 'iso-client-2' came back up; going online.
[2015-11-24 10:03:09.233301] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-iso-replicate-1: Client-quorum is met
[2015-11-24 10:03:09.233504] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-2: Server lk version = 1
[2015-11-24 10:03:09.233788] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-2: Connected to storage-client-2, attached to remote volume '/tank/storage/brick2'.
[2015-11-24 10:03:09.233806] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.233813] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-storage-client-2: 1 fds open - Delaying child_up until they are re-opened
[2015-11-24 10:03:09.234559] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-storage-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2015-11-24 10:03:09.234594] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-storage-replicate-1: Subvolume 'storage-client-2' came back up; going online.
[2015-11-24 10:03:09.234604] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-storage-replicate-1: Client-quorum is met
[2015-11-24 10:03:09.234962] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.234977] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-2: Server lk version = 1
[2015-11-24 10:03:09.235282] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-0: Connected to iso-client-0, attached to remote volume '/tank/iso/brick1'.
[2015-11-24 10:03:09.235300] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.235338] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-iso-replicate-0: Subvolume 'iso-client-0' came back up; going online.
[2015-11-24 10:03:09.235349] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-iso-replicate-0: Client-quorum is met
[2015-11-24 10:03:09.235559] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-0: Server lk version = 1
[2015-11-24 10:03:09.237483] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.239948] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.240285] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/tank/testvol/brick1'.
[2015-11-24 10:03:09.240302] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.240340] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-testvol-replicate-0: Subvolume 'testvol-client-0' came back up; going online.
[2015-11-24 10:03:09.240352] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-4: Connected to data-client-4, attached to remote volume '/tank/data/brick3'.
[2015-11-24 10:03:09.240369] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.240382] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-data-client-4: 2 fds open - Delaying child_up until they are re-opened
[2015-11-24 10:03:09.240626] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1
[2015-11-24 10:03:09.241119] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-data-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2015-11-24 10:03:09.241147] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-data-replicate-1: Subvolume 'data-client-4' came back up; going online.
[2015-11-24 10:03:09.241558] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-4: Server lk version = 1
[2015-11-24 10:03:09.242715] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:09.245015] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-2: Connected to testvol-client-2, attached to remote volume '/tank/testvol/brick2'.
[2015-11-24 10:03:09.245033] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:09.245071] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-testvol-replicate-1: Subvolume 'testvol-client-2' came back up; going online.
[2015-11-24 10:03:09.245458] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-2: Server lk version = 1
[2015-11-24 10:03:09.999399] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7df5) [0x7f0969698df5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f096ad03855] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f096ad036d9] ) 0-: received signum (15), shutting down
[2015-11-24 10:03:11.051680] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/83d53ca838047102dd70bd5a7fe42b53.socket --xlator-option *replicate*.node-uuid=5fd7c455-4eef-4e1f-9ef4-91b47748bebc)
[2015-11-24 10:03:11.062745] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-11-24 10:03:11.065699] I [graph.c:269:gf_add_cmdline_options] 0-data-replicate-1: adding option 'node-uuid' for volume 'data-replicate-1' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065720] I [graph.c:269:gf_add_cmdline_options] 0-data-replicate-0: adding option 'node-uuid' for volume 'data-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065728] I [graph.c:269:gf_add_cmdline_options] 0-export-replicate-1: adding option 'node-uuid' for volume 'export-replicate-1' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065734] I [graph.c:269:gf_add_cmdline_options] 0-export-replicate-0: adding option 'node-uuid' for volume 'export-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065749] I [graph.c:269:gf_add_cmdline_options] 0-iso-replicate-1: adding option 'node-uuid' for volume 'iso-replicate-1' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065756] I [graph.c:269:gf_add_cmdline_options] 0-iso-replicate-0: adding option 'node-uuid' for volume 'iso-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065762] I [graph.c:269:gf_add_cmdline_options] 0-meta-replicate-0: adding option 'node-uuid' for volume 'meta-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065768] I [graph.c:269:gf_add_cmdline_options] 0-storage-replicate-1: adding option 'node-uuid' for volume 'storage-replicate-1' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065774] I [graph.c:269:gf_add_cmdline_options] 0-storage-replicate-0: adding option 'node-uuid' for volume 'storage-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065780] I [graph.c:269:gf_add_cmdline_options] 0-testvol-replicate-1: adding option 'node-uuid' for volume 'testvol-replicate-1' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.065786] I [graph.c:269:gf_add_cmdline_options] 0-testvol-replicate-0: adding option 'node-uuid' for volume 'testvol-replicate-0' with value '5fd7c455-4eef-4e1f-9ef4-91b47748bebc'
[2015-11-24 10:03:11.070733] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2015-11-24 10:03:11.070776] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2015-11-24 10:03:11.070849] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2015-11-24 10:03:11.071309] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 6
[2015-11-24 10:03:11.071402] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 5
[2015-11-24 10:03:11.071499] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 7
[2015-11-24 10:03:11.071508] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 8
[2015-11-24 10:03:11.094986] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.098455] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.099307] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)
[2015-11-24 10:03:11.101294] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-2: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.104973] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.105517] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/tank/data/brick1'.
[2015-11-24 10:03:11.105534] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.105592] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-data-replicate-0: Subvolume 'data-client-0' came back up; going online.
[2015-11-24 10:03:11.105858] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-0: Server lk version = 1
[2015-11-24 10:03:11.106463] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-3: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.107197] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-2: changing port to 49154 (from 0)
[2015-11-24 10:03:11.109357] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-4: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.112310] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.112782] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-2: Connected to data-client-2, attached to remote volume '/tank/data/brick2'.
[2015-11-24 10:03:11.112796] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.113062] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-2: Server lk version = 1
[2015-11-24 10:03:11.114286] I [MSGID: 114020] [client.c:2118:notify] 0-data-client-5: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.115038] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-4: changing port to 49156 (from 0)
[2015-11-24 10:03:11.117209] I [MSGID: 114020] [client.c:2118:notify] 0-export-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.120326] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.120721] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-4: Connected to data-client-4, attached to remote volume '/tank/data/brick3'.
[2015-11-24 10:03:11.120736] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.120777] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-data-replicate-1: Subvolume 'data-client-4' came back up; going online.
[2015-11-24 10:03:11.121096] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-4: Server lk version = 1
[2015-11-24 10:03:11.122348] I [MSGID: 114020] [client.c:2118:notify] 0-export-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.123108] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-0: changing port to 49158 (from 0)
[2015-11-24 10:03:11.125470] I [MSGID: 114020] [client.c:2118:notify] 0-export-client-2: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.128719] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.129213] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-0: Connected to export-client-0, attached to remote volume '/tank/export/brick1'.
[2015-11-24 10:03:11.129228] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.129293] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-export-replicate-0: Subvolume 'export-client-0' came back up; going online.
[2015-11-24 10:03:11.129614] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-0: Server lk version = 1
[2015-11-24 10:03:11.130814] I [MSGID: 114020] [client.c:2118:notify] 0-export-client-3: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.131541] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-2: changing port to 49159 (from 0)
[2015-11-24 10:03:11.135130] I [MSGID: 114020] [client.c:2118:notify] 0-iso-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.138223] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.138666] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-2: Connected to export-client-2, attached to remote volume '/tank/export/brick2'.
[2015-11-24 10:03:11.138691] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.138735] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-export-replicate-1: Subvolume 'export-client-2' came back up; going online.
[2015-11-24 10:03:11.139088] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-2: Server lk version = 1
[2015-11-24 10:03:11.141827] I [MSGID: 114020] [client.c:2118:notify] 0-iso-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.142527] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-0: changing port to 49160 (from 0)
[2015-11-24 10:03:11.146371] I [MSGID: 114020] [client.c:2118:notify] 0-iso-client-2: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.149318] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.149731] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-0: Connected to iso-client-0, attached to remote volume '/tank/iso/brick1'.
[2015-11-24 10:03:11.149745] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.149789] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-iso-replicate-0: Subvolume 'iso-client-0' came back up; going online.
[2015-11-24 10:03:11.150093] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-0: Server lk version = 1
[2015-11-24 10:03:11.153022] I [MSGID: 114020] [client.c:2118:notify] 0-iso-client-3: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.153705] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-2: changing port to 49161 (from 0)
[2015-11-24 10:03:11.156722] I [MSGID: 114020] [client.c:2118:notify] 0-meta-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.159713] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.160154] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-2: Connected to iso-client-2, attached to remote volume '/tank/iso/brick2'.
[2015-11-24 10:03:11.160169] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.160209] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-iso-replicate-1: Subvolume 'iso-client-2' came back up; going online.
[2015-11-24 10:03:11.160523] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-2: Server lk version = 1
[2015-11-24 10:03:11.161776] I [MSGID: 114020] [client.c:2118:notify] 0-meta-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.162544] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-meta-client-0: changing port to 49162 (from 0)
[2015-11-24 10:03:11.164725] I [MSGID: 114020] [client.c:2118:notify] 0-storage-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.167713] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-meta-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.168094] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-meta-client-0: Connected to meta-client-0, attached to remote volume '/tank/meta/brick1'.
[2015-11-24 10:03:11.168109] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-meta-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.168156] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-meta-replicate-0: Subvolume 'meta-client-0' came back up; going online.
[2015-11-24 10:03:11.168432] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-meta-client-0: Server lk version = 1
[2015-11-24 10:03:11.169799] I [MSGID: 114020] [client.c:2118:notify] 0-storage-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.170578] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-0: changing port to 49163 (from 0)
[2015-11-24 10:03:11.172656] I [MSGID: 114020] [client.c:2118:notify] 0-storage-client-2: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.175663] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.176095] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-0: Connected to storage-client-0, attached to remote volume '/tank/storage/brick1'.
[2015-11-24 10:03:11.176115] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.176155] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-storage-replicate-0: Subvolume 'storage-client-0' came back up; going online.
[2015-11-24 10:03:11.176393] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-0: Server lk version = 1
[2015-11-24 10:03:11.177687] I [MSGID: 114020] [client.c:2118:notify] 0-storage-client-3: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.178410] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-2: changing port to 49164 (from 0)
[2015-11-24 10:03:11.180651] I [MSGID: 114020] [client.c:2118:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.183654] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.184033] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-2: Connected to storage-client-2, attached to remote volume '/tank/storage/brick2'.
[2015-11-24 10:03:11.184048] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.184088] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-storage-replicate-1: Subvolume 'storage-client-2' came back up; going online.
[2015-11-24 10:03:11.184387] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-2: Server lk version = 1
[2015-11-24 10:03:11.185684] I [MSGID: 114020] [client.c:2118:notify] 0-testvol-client-1: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.186404] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49165 (from 0)
[2015-11-24 10:03:11.188699] I [MSGID: 114020] [client.c:2118:notify] 0-testvol-client-2: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.191620] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.192017] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/tank/testvol/brick1'.
[2015-11-24 10:03:11.192035] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.192081] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-testvol-replicate-0: Subvolume 'testvol-client-0' came back up; going online.
[2015-11-24 10:03:11.192454] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1
[2015-11-24 10:03:11.193729] I [MSGID: 114020] [client.c:2118:notify] 0-testvol-client-3: parent translators are ready, attempting connect on transport
[2015-11-24 10:03:11.194562] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-2: changing port to 49166 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume data-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key gluster/glustershd
  6:     option client-version 3.7.6
  7:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-data-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 10
 10:     option remote-host 139.64.206.100
 11:     option remote-subvolume /tank/data/brick1
 12:     option transport-type socket
 13:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
 14:     option password f8989d3c-001e-42b7-9862-e978b9996195
 15:     option filter-O_DIRECT enable
 16:     option event-threads 8
 17:     option client-bind-insecure on
 18: end-volume
 19:
 20: volume data-client-1
 21:     type protocol/client
 22:     option ping-timeout 10
 23:     option remote-host 139.64.206.101
 24:     option remote-subvolume /tank/data/brick1
 25:     option transport-type socket
 26:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
 27:     option password f8989d3c-001e-42b7-9862-e978b9996195
 28:     option filter-O_DIRECT enable
 29:     option event-threads 8
 30:     option client-bind-insecure on
 31: end-volume
 32:
 33: volume data-client-2
 34:     type protocol/client
 35:     option clnt-lk-version 1
 36:     option volfile-checksum 0
 37:     option volfile-key gluster/glustershd
 38:     option client-version 3.7.6
 39:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-data-client-2-0-0
 40:     option fops-version 1298437
 41:     option ping-timeout 10
 42:     option remote-host 139.64.206.100
 43:     option remote-subvolume /tank/data/brick2
 44:     option transport-type socket
 45:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
 46:     option password f8989d3c-001e-42b7-9862-e978b9996195
 47:     option filter-O_DIRECT enable
 48:     option event-threads 8
 49:     option client-bind-insecure on
 50: end-volume
 51:
 52: volume data-replicate-0
 53:     type cluster/replicate
 54:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
 55:     option background-self-heal-count 0
 56:     option metadata-self-heal on
 57:     option data-self-heal on
 58:     option entry-self-heal on
 59:     option self-heal-daemon enable
 60:     option eager-lock enable
 61:     option quorum-type auto
 62:     option iam-self-heal-daemon yes
 63:     subvolumes data-client-0 data-client-1 data-client-2
 64: end-volume
 65:
 66: volume data-client-3
 67:     type protocol/client
 68:     option ping-timeout 10
 69:     option remote-host 139.64.206.101
 70:     option remote-subvolume /tank/data/brick2
 71:     option transport-type socket
 72:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
 73:     option password f8989d3c-001e-42b7-9862-e978b9996195
 74:     option filter-O_DIRECT enable
 75:     option event-threads 8
 76:     option client-bind-insecure on
 77: end-volume
 78:
 79: volume data-client-4
 80:     type protocol/client
 81:     option clnt-lk-version 1
 82:     option volfile-checksum 0
 83:     option volfile-key gluster/glustershd
 84:     option client-version 3.7.6
 85:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-data-client-4-0-0
 86:     option fops-version 1298437
 87:     option ping-timeout 10
 88:     option remote-host 139.64.206.100
 89:     option remote-subvolume /tank/data/brick3
 90:     option transport-type socket
 91:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
 92:     option password f8989d3c-001e-42b7-9862-e978b9996195
 93:     option filter-O_DIRECT enable
 94:     option event-threads 8
 95:     option client-bind-insecure on
 96: end-volume
 97:
 98: volume data-client-5
 99:     type protocol/client
100:     option ping-timeout 10
101:     option remote-host 139.64.206.101
102:     option remote-subvolume /tank/data/brick3
103:     option transport-type socket
104:     option username 0db4e7d6-ba62-4fa5-9d18-0bf3f141b0b9
105:     option password f8989d3c-001e-42b7-9862-e978b9996195
106:     option filter-O_DIRECT enable
107:     option event-threads 8
108:     option client-bind-insecure on
109: end-volume
110:
111: volume data-replicate-1
112:     type cluster/replicate
113:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
114:     option background-self-heal-count 0
115:     option metadata-self-heal on
116:     option data-self-heal on
117:     option entry-self-heal on
118:     option self-heal-daemon enable
119:     option eager-lock enable
120:     option quorum-type auto
121:     option iam-self-heal-daemon yes
122:     subvolumes data-client-3 data-client-4 data-client-5
123: end-volume
124:
125: volume export-client-0
126:     type protocol/client
127:     option clnt-lk-version 1
128:     option volfile-checksum 0
129:     option volfile-key gluster/glustershd
130:     option client-version 3.7.6
131:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-export-client-0-0-0
132:     option fops-version 1298437
133:     option ping-timeout 42
134:     option remote-host 139.64.206.100
135:     option remote-subvolume /tank/export/brick1
136:     option transport-type socket
137:     option username ce881b02-96b2-4ebb-bec8-12693a9a4e59
138:     option password 21d3771f-4a01-44f1-870a-242da85b1778
139:     option filter-O_DIRECT enable
140:     option client-bind-insecure on
141: end-volume
142:
143: volume export-client-1
144:     type protocol/client
145:     option ping-timeout 42
146:     option remote-host 139.64.206.101
147:     option remote-subvolume /tank/export/brick1
148:     option transport-type socket
149:     option username ce881b02-96b2-4ebb-bec8-12693a9a4e59
150:     option password 21d3771f-4a01-44f1-870a-242da85b1778
151:     option filter-O_DIRECT enable
152:     option client-bind-insecure on
153: end-volume
154:
155: volume export-replicate-0
156:     type cluster/replicate
157:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
158:     option background-self-heal-count 0
159:     option metadata-self-heal on
160:     option data-self-heal on
161:     option entry-self-heal on
162:     option self-heal-daemon enable
163:     option eager-lock enable
164:     option quorum-type auto
165:     option iam-self-heal-daemon yes
166:     subvolumes export-client-0 export-client-1
167: end-volume
168:
169: volume export-client-2
170:     type protocol/client
171:     option clnt-lk-version 1
172:     option volfile-checksum 0
173:     option volfile-key gluster/glustershd
174:     option client-version 3.7.6
175:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-export-client-2-0-0
176:     option fops-version 1298437
177:     option ping-timeout 42
178:     option remote-host 139.64.206.100
179:     option remote-subvolume /tank/export/brick2
180:     option transport-type socket
181:     option username ce881b02-96b2-4ebb-bec8-12693a9a4e59
182:     option password 21d3771f-4a01-44f1-870a-242da85b1778
183:     option filter-O_DIRECT enable
184:     option client-bind-insecure on
185: end-volume
186:
187: volume export-client-3
188:     type protocol/client
189:     option ping-timeout 42
190:     option remote-host 139.64.206.101
191:     option remote-subvolume /tank/export/brick2
192:     option transport-type socket
193:     option username ce881b02-96b2-4ebb-bec8-12693a9a4e59
194:     option password 21d3771f-4a01-44f1-870a-242da85b1778
195:     option filter-O_DIRECT enable
196:     option client-bind-insecure on
197: end-volume
198:
199: volume export-replicate-1
200:     type cluster/replicate
201:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
202:     option background-self-heal-count 0
203:     option metadata-self-heal on
204:     option data-self-heal on
205:     option entry-self-heal on
206:     option self-heal-daemon enable
207:     option eager-lock enable
208:     option quorum-type auto
209:     option iam-self-heal-daemon yes
210:     subvolumes export-client-2 export-client-3
211: end-volume
212:
213: volume iso-client-0
214:     type protocol/client
215:     option clnt-lk-version 1
216:     option volfile-checksum 0
217:     option volfile-key gluster/glustershd
218:     option client-version 3.7.6
219:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-iso-client-0-0-0
220:     option fops-version 1298437
221:     option ping-timeout 42
222:     option remote-host 139.64.206.100
223:     option remote-subvolume /tank/iso/brick1
224:     option transport-type socket
225:     option username f9eadf47-a0a5-4904-96bd-992e667c1213
226:     option password a5123f20-2818-4512-8aec-56a4ad54d8b0
227:     option filter-O_DIRECT enable
228:     option client-bind-insecure on
229: end-volume
230:
231: volume iso-client-1
232:     type protocol/client
233:     option ping-timeout 42
234:     option remote-host 139.64.206.101
235:     option remote-subvolume /tank/iso/brick1
236:     option transport-type socket
237:     option username f9eadf47-a0a5-4904-96bd-992e667c1213
238:     option password a5123f20-2818-4512-8aec-56a4ad54d8b0
239:     option filter-O_DIRECT enable
240:     option client-bind-insecure on
241: end-volume
242:
243: volume iso-replicate-0
244:     type cluster/replicate
245:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
246:     option background-self-heal-count 0
247:     option metadata-self-heal on
248:     option data-self-heal on
249:     option entry-self-heal on
250:     option self-heal-daemon enable
251:     option eager-lock enable
252:     option quorum-type auto
253:     option iam-self-heal-daemon yes
254:     subvolumes iso-client-0 iso-client-1
255: end-volume
256:
257: volume iso-client-2
258:     type protocol/client
259:     option clnt-lk-version 1
260:     option volfile-checksum 0
261:     option volfile-key gluster/glustershd
262:     option client-version 3.7.6
263:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-iso-client-2-0-0
264:     option fops-version 1298437
265:     option ping-timeout 42
266:     option remote-host 139.64.206.100
267:     option remote-subvolume /tank/iso/brick2
268:     option transport-type socket
269:     option username f9eadf47-a0a5-4904-96bd-992e667c1213
270:     option password a5123f20-2818-4512-8aec-56a4ad54d8b0
271:     option filter-O_DIRECT enable
272:     option client-bind-insecure on
273: end-volume
274:
275: volume iso-client-3
276:     type protocol/client
277:     option ping-timeout 42
278:     option remote-host 139.64.206.101
279:     option remote-subvolume /tank/iso/brick2
280:     option transport-type socket
281:     option username f9eadf47-a0a5-4904-96bd-992e667c1213
282:     option password a5123f20-2818-4512-8aec-56a4ad54d8b0
283:     option filter-O_DIRECT enable
284:     option client-bind-insecure on
285: end-volume
286:
287: volume iso-replicate-1
288:     type cluster/replicate
289:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
290:     option background-self-heal-count 0
291:     option metadata-self-heal on
292:     option data-self-heal on
293:     option entry-self-heal on
294:     option self-heal-daemon enable
295:     option eager-lock enable
296:     option quorum-type auto
297:     option iam-self-heal-daemon yes
298:     subvolumes iso-client-2 iso-client-3
299: end-volume
300:
301: volume meta-client-0
302:     type protocol/client
303:     option clnt-lk-version 1
304:     option volfile-checksum 0
305:     option volfile-key gluster/glustershd
306:     option client-version 3.7.6
307:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-meta-client-0-0-0
308:     option fops-version 1298437
309:     option ping-timeout 42
310:     option remote-host glusternode1
311:     option remote-subvolume /tank/meta/brick1
312:     option transport-type socket
313:     option username b48ab853-3cc0-41d1-9355-537516e68c02
314:     option password 272b1b9d-8155-480a-827c-184b2de68138
315: end-volume
316:
317: volume meta-client-1
318:     type protocol/client
319:     option ping-timeout 42
320:     option remote-host glusternode2
321:     option remote-subvolume /tank/meta/brick1
322:     option transport-type socket
323:     option username b48ab853-3cc0-41d1-9355-537516e68c02
324:     option password 272b1b9d-8155-480a-827c-184b2de68138
325: end-volume
326:
327: volume meta-replicate-0
328:     type cluster/replicate
329:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
330:     option background-self-heal-count 0
331:     option metadata-self-heal on
332:     option data-self-heal on
333:     option entry-self-heal on
334:     option self-heal-daemon enable
335:     option iam-self-heal-daemon yes
336:     subvolumes meta-client-0 meta-client-1
337: end-volume
338:
339: volume storage-client-0
340:     type protocol/client
341:     option clnt-lk-version 1
342:     option volfile-checksum 0
343:     option volfile-key gluster/glustershd
344:     option client-version 3.7.6
345:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-storage-client-0-0-0
346:     option fops-version 1298437
347:     option ping-timeout 42
348:     option remote-host 139.64.206.100
349:     option remote-subvolume /tank/storage/brick1
350:     option transport-type socket
351:     option username 2a21e863-2015-4aab-8ef7-e107add864e5
352:     option password 7b8ec572-4bbf-498b-9cf7-895617fe5bcd
353:     option filter-O_DIRECT enable
354: end-volume
355:
356: volume storage-client-1
357:     type protocol/client
358:     option ping-timeout 42
359:     option remote-host 139.64.206.101
360:     option remote-subvolume /tank/storage/brick1
361:     option transport-type socket
362:     option username 2a21e863-2015-4aab-8ef7-e107add864e5
363:     option password 7b8ec572-4bbf-498b-9cf7-895617fe5bcd
364:     option filter-O_DIRECT enable
365: end-volume
366:
367: volume storage-replicate-0
368:     type cluster/replicate
369:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
370:     option background-self-heal-count 0
371:     option metadata-self-heal on
372:     option data-self-heal on
373:     option entry-self-heal on
374:     option self-heal-daemon enable
375:     option eager-lock enable
376:     option quorum-type auto
377:     option iam-self-heal-daemon yes
378:     subvolumes storage-client-0 storage-client-1
379: end-volume
380:
381: volume storage-client-2
382:     type protocol/client
383:     option clnt-lk-version 1
384:     option volfile-checksum 0
385:     option volfile-key gluster/glustershd
386:     option client-version 3.7.6
387:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-storage-client-2-0-0
388:     option fops-version 1298437
389:     option ping-timeout 42
390:     option remote-host 139.64.206.100
391:     option remote-subvolume /tank/storage/brick2
392:     option transport-type socket
393:     option username 2a21e863-2015-4aab-8ef7-e107add864e5
394:     option password 7b8ec572-4bbf-498b-9cf7-895617fe5bcd
395:     option filter-O_DIRECT enable
396: end-volume
397:
398: volume storage-client-3
399:     type protocol/client
400:     option ping-timeout 42
401:     option remote-host 139.64.206.101
402:     option remote-subvolume /tank/storage/brick2
403:     option transport-type socket
404:     option username 2a21e863-2015-4aab-8ef7-e107add864e5
405:     option password 7b8ec572-4bbf-498b-9cf7-895617fe5bcd
406:     option filter-O_DIRECT enable
407: end-volume
408:
409: volume storage-replicate-1
410:     type cluster/replicate
411:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
412:     option background-self-heal-count 0
413:     option metadata-self-heal on
414:     option data-self-heal on
415:     option entry-self-heal on
416:     option self-heal-daemon enable
417:     option eager-lock enable
418:     option quorum-type auto
419:     option iam-self-heal-daemon yes
420:     subvolumes storage-client-2 storage-client-3
421: end-volume
422:
423: volume testvol-client-0
424:     type protocol/client
425:     option clnt-lk-version 1
426:     option volfile-checksum 0
427:     option volfile-key gluster/glustershd
428:     option client-version 3.7.6
429:     option process-uuid glusternode2.bodden-kliniken.net-22100-2015/11/24-10:03:11:48484-testvol-client-0-0-0
430:     option fops-version 1298437
431:     option ping-timeout 42
432:     option remote-host 139.64.206.100
433:     option remote-subvolume /tank/testvol/brick1
434:     option transport-type socket
435:     option username 74251182-d521-4291-90ce-3b214e182f1a
436:     option password 97d8c3f1-684e-4c63-95a3-08101878ba68
437:     option filter-O_DIRECT enable
438:     option event-threads 8
439:     option client-bind-insecure on
440: end-volume
441:
442: volume testvol-client-1
443:     type protocol/client
444:     option ping-timeout 42
445:     option remote-host 139.64.206.101
446:     option remote-subvolume /tank/testvol/brick1
447:     option transport-type socket
448:     option username 74251182-d521-4291-90ce-3b214e182f1a
449:     option password 97d8c3f1-684e-4c63-95a3-08101878ba68
450:     option filter-O_DIRECT enable
451:     option event-threads 8
452:     option client-bind-insecure on
453: end-volume
454:
455: volume testvol-replicate-0
456:     type cluster/replicate
457:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
458:     option background-self-heal-count 0
459:     option metadata-self-heal on
460:     option data-self-heal on
461:     option entry-self-heal on
462:     option self-heal-daemon enable
463:     option iam-self-heal-daemon yes
464:     subvolumes testvol-client-0 testvol-client-1
465: end-volume
466:
467: volume testvol-client-2
468:     type protocol/client
469:     option ping-timeout 42
470:     option remote-host 139.64.206.100
471:     option remote-subvolume /tank/testvol/brick2
472:     option transport-type socket
473:     option username 74251182-d521-4291-90ce-3b214e182f1a
474:     option password 97d8c3f1-684e-4c63-95a3-08101878ba68
475:     option filter-O_DIRECT enable
476:     option event-threads 8
477:     option client-bind-insecure on
478: end-volume
479:
480: volume testvol-client-3
481:     type protocol/client
482:     option ping-timeout 42
483:     option remote-host 139.64.206.101
484:     option remote-subvolume /tank/testvol/brick2
485:     option transport-type socket
486:     option username 74251182-d521-4291-90ce-3b214e182f1a
487:     option password 97d8c3f1-684e-4c63-95a3-08101878ba68
488:     option filter-O_DIRECT enable
489:     option event-threads 8
490:     option client-bind-insecure on
491: end-volume
492:
493: volume testvol-replicate-1
494:     type cluster/replicate
495:     option node-uuid 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
496:     option background-self-heal-count 0
497:     option metadata-self-heal on
498:     option data-self-heal on
499:     option entry-self-heal on
500:     option self-heal-daemon enable
501:     option iam-self-heal-daemon yes
502:     subvolumes testvol-client-2 testvol-client-3
503: end-volume
504:
505: volume glustershd
506:     type debug/io-stats
507:     subvolumes data-replicate-0 data-replicate-1 export-replicate-0 export-replicate-1 iso-replicate-0 iso-replicate-1 meta-replicate-0 storage-replicate-0 storage-replicate-1 testvol-replicate-0 testvol-replicate-1
508: end-volume
509:
+------------------------------------------------------------------------------+
[2015-11-24 10:03:11.199929] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.200306] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-2: Connected to testvol-client-2, attached to remote volume '/tank/testvol/brick2'.
[2015-11-24 10:03:11.200335] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.200404] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-testvol-replicate-1: Subvolume 'testvol-client-2' came back up; going online.
[2015-11-24 10:03:11.200717] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-2: Server lk version = 1
[2015-11-24 10:03:11.278599] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-3: changing port to 50198 (from 0)
[2015-11-24 10:03:11.281995] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.284393] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-3: Connected to data-client-3, attached to remote volume '/tank/data/brick2'.
[2015-11-24 10:03:11.284418] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.284577] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-3: Server lk version = 1
[2015-11-24 10:03:11.293561] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-1: changing port to 50190 (from 0)
[2015-11-24 10:03:11.296937] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.299382] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-1: Connected to data-client-1, attached to remote volume '/tank/data/brick1'.
[2015-11-24 10:03:11.299402] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.299593] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-1: Server lk version = 1
[2015-11-24 10:03:11.312021] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-3: changing port to 50195 (from 0)
[2015-11-24 10:03:11.312023] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-5: changing port to 50203 (from 0)
[2015-11-24 10:03:11.312101] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-1: changing port to 50194 (from 0)
[2015-11-24 10:03:11.312108] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-1: changing port to 50196 (from 0)
[2015-11-24 10:03:11.312156] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-3: changing port to 50197 (from 0)
[2015-11-24 10:03:11.312225] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-meta-client-1: changing port to 50200 (from 0)
[2015-11-24 10:03:11.312312] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-1: changing port to 50186 (from 0)
[2015-11-24 10:03:11.312362] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-3: changing port to 50187 (from 0)
[2015-11-24 10:03:11.312372] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-1: changing port to 50188 (from 0)
[2015-11-24 10:03:11.312414] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-3: changing port to 50189 (from 0)
[2015-11-24 10:03:11.320669] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.320734] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.320783] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.323203] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-5: Connected to data-client-5, attached to remote volume '/tank/data/brick3'.
[2015-11-24 10:03:11.323228] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-5: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.323250] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.323808] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-5: Server lk version = 1
[2015-11-24 10:03:11.323930] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-3: Connected to iso-client-3, attached to remote volume '/tank/iso/brick2'.
[2015-11-24 10:03:11.323935] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-3: Connected to export-client-3, attached to remote volume '/tank/export/brick2'.
[2015-11-24 10:03:11.323943] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.323958] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.324171] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-3: Server lk version = 1
[2015-11-24 10:03:11.324902] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-3: Server lk version = 1
[2015-11-24 10:03:11.325826] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.326048] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-1: Connected to iso-client-1, attached to remote volume '/tank/iso/brick1'.
[2015-11-24 10:03:11.326063] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.326771] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-1: Server lk version = 1
[2015-11-24 10:03:11.328793] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-1: Connected to storage-client-1, attached to remote volume '/tank/storage/brick1'.
[2015-11-24 10:03:11.328831] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.329312] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-1: Server lk version = 1
[2015-11-24 10:03:11.329960] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.332301] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-3: Connected to storage-client-3, attached to remote volume '/tank/storage/brick2'.
[2015-11-24 10:03:11.332321] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.332459] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.332467] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-3: Server lk version = 1
[2015-11-24 10:03:11.332671] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-1: Connected to export-client-1, attached to remote volume '/tank/export/brick1'.
[2015-11-24 10:03:11.332685] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.332829] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-1: Server lk version = 1
[2015-11-24 10:03:11.335772] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-meta-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.338406] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-meta-client-1: Connected to meta-client-1, attached to remote volume '/tank/meta/brick1'.
[2015-11-24 10:03:11.338444] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-meta-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.339074] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-meta-client-1: Server lk version = 1
[2015-11-24 10:03:11.340359] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.342557] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-1: Connected to testvol-client-1, attached to remote volume '/tank/testvol/brick1'.
[2015-11-24 10:03:11.342576] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.342754] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-1: Server lk version = 1
[2015-11-24 10:03:11.342932] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:03:11.345395] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-3: Connected to testvol-client-3, attached to remote volume '/tank/testvol/brick2'.
[2015-11-24 10:03:11.345412] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:03:11.345706] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-3: Server lk version = 1
[2015-11-24 10:03:11.354945] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-1: performing entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a
[2015-11-24 10:03:11.356763] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:03:11.356929] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-0: performing entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a
[2015-11-24 10:03:11.361702] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:03:11.363920] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:03:11.367199] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:03:11.371892] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-1: Completed entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a. source=-1 sinks=0 1
[2015-11-24 10:03:11.381078] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-0: Completed entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a. source=-1 sinks=0 1
[2015-11-24 10:03:11.397613] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-0: Completed data selfheal on b4679dab-b23e-4319-8878-c8385b2c2749. source=1 sinks=0
[2015-11-24 10:03:11.433677] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-1: performing entry selfheal on 00000000-0000-0000-0000-000000000001
[2015-11-24 10:03:11.464718] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-1: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. source=-1 sinks=0 1
[2015-11-24 10:03:11.476700] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-1: Completed data selfheal on 8753e655-fe4b-4643-aa57-a66a9076f93d. source=1 sinks=0
[2015-11-24 10:04:42.621823] W [socket.c:588:__socket_rwv] 0-iso-client-2: readv on 139.64.206.100:49161 failed (No data available)
[2015-11-24 10:04:42.621869] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-2: disconnected from iso-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.621905] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-iso-replicate-1: Client-quorum is not met
[2015-11-24 10:04:42.623378] W [socket.c:588:__socket_rwv] 0-export-client-2: readv on 139.64.206.100:49159 failed (No data available)
[2015-11-24 10:04:42.623415] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-2: disconnected from export-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.623440] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-export-replicate-1: Client-quorum is not met
[2015-11-24 10:04:42.623533] W [socket.c:588:__socket_rwv] 0-meta-client-0: readv on 139.64.206.100:49162 failed (No data available)
[2015-11-24 10:04:42.623555] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-meta-client-0: disconnected from meta-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.623645] W [socket.c:588:__socket_rwv] 0-iso-client-0: readv on 139.64.206.100:49160 failed (No data available)
[2015-11-24 10:04:42.623672] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-0: disconnected from iso-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.623713] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-iso-replicate-0: Client-quorum is not met
[2015-11-24 10:04:42.624365] W [socket.c:588:__socket_rwv] 0-export-client-0: readv on 139.64.206.100:49158 failed (No data available)
[2015-11-24 10:04:42.624451] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-0: disconnected from export-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.624479] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-export-replicate-0: Client-quorum is not met
[2015-11-24 10:04:42.624781] W [socket.c:588:__socket_rwv] 0-testvol-client-0: readv on 139.64.206.100:49165 failed (No data available)
[2015-11-24 10:04:42.624780] W [socket.c:588:__socket_rwv] 0-testvol-client-2: readv on 139.64.206.100:49166 failed (No data available)
[2015-11-24 10:04:42.624786] W [socket.c:588:__socket_rwv] 0-storage-client-0: readv on 139.64.206.100:49163 failed (No data available)
[2015-11-24 10:04:42.624867] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.624876] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-2: disconnected from testvol-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.624887] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-0: disconnected from storage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.624951] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-storage-replicate-0: Client-quorum is not met
[2015-11-24 10:04:42.629737] W [socket.c:588:__socket_rwv] 0-data-client-2: readv on 139.64.206.100:49154 failed (No data available)
[2015-11-24 10:04:42.629798] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-2: disconnected from data-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.629873] W [socket.c:588:__socket_rwv] 0-data-client-0: readv on 139.64.206.100:49152 failed (No data available)
[2015-11-24 10:04:42.629911] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-0: disconnected from data-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.629922] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-data-replicate-0: Client-quorum is not met
[2015-11-24 10:04:42.641264] W [socket.c:588:__socket_rwv] 0-storage-client-2: readv on 139.64.206.100:49164 failed (No data available)
[2015-11-24 10:04:42.641675] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f89bcacba82] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f89bc896a3e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f89bc896b4e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f89bc8984da] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f89bc898d08] ))))) 0-storage-client-2: forced unwinding frame type(GlusterFS 3.3) op(RCHECKSUM(37)) called at 2015-11-24 10:04:42.563897 (xid=0x188a2)
[2015-11-24 10:04:42.641692] W [MSGID: 114031] [client-rpc-fops.c:2454:client3_3_rchecksum_cbk] 0-storage-client-2: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.641711] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-2: disconnected from storage-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.641720] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-storage-replicate-1: Client-quorum is not met
[2015-11-24 10:04:42.641784] W [MSGID: 114031] [client-rpc-fops.c:3047:client3_3_readv_cbk] 0-storage-client-2: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.641824] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-storage-client-2: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.707564] W [socket.c:588:__socket_rwv] 0-data-client-4: readv on 139.64.206.100:49156 failed (No data available)
[2015-11-24 10:04:42.707849] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f89bcacba82] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f89bc896a3e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f89bc896b4e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f89bc8984da] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f89bc898d08] ))))) 0-data-client-4: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2015-11-24 10:04:42.614758 (xid=0x28df)
The message "E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-storage-client-2: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2015-11-24 10:04:42.641824] and [2015-11-24 10:04:42.642784]
[2015-11-24 10:04:42.707865] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.708045] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f89bcacba82] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f89bc896a3e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f89bc896b4e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f89bc8984da] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f89bc898d08] ))))) 0-data-client-4: forced unwinding frame type(GlusterFS 3.3) op(RCHECKSUM(37)) called at 2015-11-24 10:04:42.614505 (xid=0x28de)
[2015-11-24 10:04:42.708068] W [MSGID: 114031] [client-rpc-fops.c:2454:client3_3_rchecksum_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.710680] I [socket.c:3308:socket_submit_request] 0-data-client-4: not connected (priv->connected = 0)
[2015-11-24 10:04:42.710694] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-data-client-4: failed to submit rpc-request (XID: 0x28e0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 37) to rpc-transport (data-client-4)
[2015-11-24 10:04:42.710711] W [MSGID: 114031] [client-rpc-fops.c:2454:client3_3_rchecksum_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.710722] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-4: disconnected from data-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:42.710775] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-data-client-4: failed to submit rpc-request (XID: 0x28e1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (data-client-4)
[2015-11-24 10:04:42.710801] W [MSGID: 114031] [client-rpc-fops.c:904:client3_3_writev_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.710834] W [MSGID: 114029] [client-rpc-fops.c:4412:client3_3_writev] 0-data-client-4: failed to send the fop
[2015-11-24 10:04:42.710883] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.710955] E [socket.c:2278:socket_connect_finish] 0-data-client-4: connection to 139.64.206.100:24007 failed (Connection refused)
[2015-11-24 10:04:42.715863] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-data-client-4: failed to submit rpc-request (XID: 0x28e2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (data-client-4)
The message "E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2015-11-24 10:04:42.710883] and [2015-11-24 10:04:42.712312]
[2015-11-24 10:04:42.715884] W [MSGID: 114031] [client-rpc-fops.c:904:client3_3_writev_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.715917] W [MSGID: 114029] [client-rpc-fops.c:4412:client3_3_writev] 0-data-client-4: failed to send the fop
[2015-11-24 10:04:42.716327] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:42.716599] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-4: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:04:53.627528] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-iso-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.627631] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-2: disconnected from iso-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.627758] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-export-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.627798] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-2: disconnected from export-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.628041] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-meta-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.628087] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-meta-client-0: disconnected from meta-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.628258] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-iso-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.628337] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-0: disconnected from iso-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.628586] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvol-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.628637] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-2: disconnected from testvol-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.628660] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-export-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.628702] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-0: disconnected from export-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.628849] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvol-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.628891] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.629125] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-storage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.629160] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-0: disconnected from storage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.629309] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.629383] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-2: disconnected from data-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.629390] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.629431] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-0: disconnected from data-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.629967] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-storage-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.630002] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:04:53.630026] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-2: disconnected from storage-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:04:53.630043] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-4: disconnected from data-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:09:39.010743] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)
[2015-11-24 10:09:39.010869] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-4: changing port to 49156 (from 0)
[2015-11-24 10:09:39.010967] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-data-client-2: changing port to 49154 (from 0)
[2015-11-24 10:09:39.014084] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:39.014524] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/tank/data/brick1'.
[2015-11-24 10:09:39.014542] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:39.014583] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-data-replicate-0: Client-quorum is met
[2015-11-24 10:09:39.014877] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-0: Server lk version = 1
[2015-11-24 10:09:39.017014] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:39.017537] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-4: Connected to data-client-4, attached to remote volume '/tank/data/brick3'.
[2015-11-24 10:09:39.017558] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:39.017569] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-data-client-4: 2 fds open - Delaying child_up until they are re-opened
[2015-11-24 10:09:39.018316] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-data-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2015-11-24 10:09:39.018748] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-4: Server lk version = 1
[2015-11-24 10:09:39.020127] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-data-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:39.020523] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-data-client-2: Connected to data-client-2, attached to remote volume '/tank/data/brick2'.
[2015-11-24 10:09:39.020542] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-data-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:39.020776] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-data-client-2: Server lk version = 1
[2015-11-24 10:09:41.271095] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-2: changing port to 49161 (from 0)
[2015-11-24 10:09:41.274114] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-2: changing port to 49159 (from 0)
[2015-11-24 10:09:41.276728] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.277135] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-2: Connected to iso-client-2, attached to remote volume '/tank/iso/brick2'.
[2015-11-24 10:09:41.277152] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.277198] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-iso-replicate-1: Client-quorum is met
[2015-11-24 10:09:41.277429] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-2: Server lk version = 1
[2015-11-24 10:09:41.279458] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-meta-client-0: changing port to 49162 (from 0)
[2015-11-24 10:09:41.282132] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.282560] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-2: Connected to export-client-2, attached to remote volume '/tank/export/brick2'.
[2015-11-24 10:09:41.282598] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.282641] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-export-replicate-1: Client-quorum is met
[2015-11-24 10:09:41.282872] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-2: Server lk version = 1
[2015-11-24 10:09:41.284959] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-iso-client-0: changing port to 49160 (from 0)
[2015-11-24 10:09:41.287560] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-meta-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.288035] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-meta-client-0: Connected to meta-client-0, attached to remote volume '/tank/meta/brick1'.
[2015-11-24 10:09:41.288051] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-meta-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.288295] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-meta-client-0: Server lk version = 1
[2015-11-24 10:09:41.290876] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-export-client-0: changing port to 49158 (from 0)
[2015-11-24 10:09:41.293383] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-iso-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.293754] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-iso-client-0: Connected to iso-client-0, attached to remote volume '/tank/iso/brick1'.
[2015-11-24 10:09:41.293771] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-iso-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.293813] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-iso-replicate-0: Client-quorum is met
[2015-11-24 10:09:41.294000] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-iso-client-0: Server lk version = 1
[2015-11-24 10:09:41.296687] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-2: changing port to 49166 (from 0)
[2015-11-24 10:09:41.299101] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-export-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.299498] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-export-client-0: Connected to export-client-0, attached to remote volume '/tank/export/brick1'.
[2015-11-24 10:09:41.299532] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-export-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.299572] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-export-replicate-0: Client-quorum is met
[2015-11-24 10:09:41.300019] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-export-client-0: Server lk version = 1
[2015-11-24 10:09:41.301807] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49165 (from 0)
[2015-11-24 10:09:41.304337] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.304704] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-2: Connected to testvol-client-2, attached to remote volume '/tank/testvol/brick2'.
[2015-11-24 10:09:41.304719] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.304933] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-2: Server lk version = 1
[2015-11-24 10:09:41.307538] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-0: changing port to 49163 (from 0)
[2015-11-24 10:09:41.309747] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.310169] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/tank/testvol/brick1'.
[2015-11-24 10:09:41.310191] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.310518] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1
[2015-11-24 10:09:41.312807] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-storage-client-2: changing port to 49164 (from 0)
[2015-11-24 10:09:41.315065] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.315394] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-0: Connected to storage-client-0, attached to remote volume '/tank/storage/brick1'.
[2015-11-24 10:09:41.315414] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.315452] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-storage-replicate-0: Client-quorum is met
[2015-11-24 10:09:41.315674] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-0: Server lk version = 1
[2015-11-24 10:09:41.315816] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-1: performing entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a
[2015-11-24 10:09:41.318028] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-storage-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-11-24 10:09:41.318372] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-storage-client-2: Connected to storage-client-2, attached to remote volume '/tank/storage/brick2'.
[2015-11-24 10:09:41.318403] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-storage-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-11-24 10:09:41.318412] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-storage-client-2: 1 fds open - Delaying child_up until they are re-opened
[2015-11-24 10:09:41.318868] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-storage-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2015-11-24 10:09:41.318896] I [MSGID: 108002] [afr-common.c:3920:afr_notify] 0-storage-replicate-1: Client-quorum is met
[2015-11-24 10:09:41.319222] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-storage-client-2: Server lk version = 1
[2015-11-24 10:09:41.321202] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:09:41.326366] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:09:41.340930] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-0: performing entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a
[2015-11-24 10:09:41.347994] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:09:41.365555] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-0: Completed entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a. source=1 sinks=0
[2015-11-24 10:09:41.367140] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-1: performing entry selfheal on 00000000-0000-0000-0000-000000000001
[2015-11-24 10:09:41.403337] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-1: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. source=-1 sinks=0 1
[2015-11-24 10:09:41.416151] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-testvol-replicate-1: Completed data selfheal on 8753e655-fe4b-4643-aa57-a66a9076f93d. source=1 sinks=0
[2015-11-24 10:09:42.466126] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-testvol-replicate-1: performing entry selfheal on a98a32d8-f775-4ce0-aa2a-9cca6dce545a
[2015-11-24 10:09:42.547229] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:09:42.618982] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-testvol-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-11-24 10:17:46.093367] W [socket.c:588:__socket_rwv] 0-data-client-1: readv on 139.64.206.101:50190 failed (No data available)
[2015-11-24 10:17:46.093422] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-1: disconnected from data-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.095118] W [socket.c:588:__socket_rwv] 0-export-client-1: readv on 139.64.206.101:50194 failed (No data available)
[2015-11-24 10:17:46.095126] W [socket.c:588:__socket_rwv] 0-storage-client-3: readv on 139.64.206.101:50187 failed (No data available)
[2015-11-24 10:17:46.095166] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-1: disconnected from export-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.095178] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-3: disconnected from storage-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.096616] W [socket.c:588:__socket_rwv] 0-data-client-5: readv on 139.64.206.101:50203 failed (No data available)
[2015-11-24 10:17:46.096649] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-5: disconnected from data-client-5. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.097428] W [socket.c:588:__socket_rwv] 0-iso-client-3: readv on 139.64.206.101:50197 failed (No data available)
[2015-11-24 10:17:46.097457] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-3: disconnected from iso-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.097559] W [socket.c:588:__socket_rwv] 0-export-client-3: readv on 139.64.206.101:50195 failed (No data available)
[2015-11-24 10:17:46.097581] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-3: disconnected from export-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100106] W [socket.c:588:__socket_rwv] 0-data-client-3: readv on 139.64.206.101:50198 failed (No data available)
[2015-11-24 10:17:46.100141] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-3: disconnected from data-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100156] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-data-replicate-1: Client-quorum is not met
[2015-11-24 10:17:46.100585] W [socket.c:588:__socket_rwv] 0-storage-client-1: readv on 139.64.206.101:50186 failed (No data available)
[2015-11-24 10:17:46.100628] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-1: disconnected from storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100790] W [socket.c:588:__socket_rwv] 0-meta-client-1: readv on 139.64.206.101:50200 failed (No data available)
[2015-11-24 10:17:46.100837] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-meta-client-1: disconnected from meta-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100793] W [socket.c:588:__socket_rwv] 0-testvol-client-1: readv on 139.64.206.101:50188 failed (No data available)
[2015-11-24 10:17:46.100923] W [socket.c:588:__socket_rwv] 0-iso-client-1: readv on 139.64.206.101:50196 failed (No data available)
[2015-11-24 10:17:46.100933] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-1: disconnected from testvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100958] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-1: disconnected from iso-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.100962] W [socket.c:588:__socket_rwv] 0-testvol-client-3: readv on 139.64.206.101:50189 failed (No data available)
[2015-11-24 10:17:46.100990] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-3: disconnected from testvol-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:46.183261] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.183316] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.183554] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.183604] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.184463] W [MSGID: 114061] [client-rpc-fops.c:5824:client3_3_rchecksum] 0-data-client-3:  (4afb616e-9c48-4b47-9a17-c0ceaeaf0ecb) remote_fd is -1. EBADFD [File descriptor in bad state]
[2015-11-24 10:17:46.242089] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242093] W [MSGID: 114031] [client-rpc-fops.c:3047:client3_3_readv_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242200] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242671] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242680] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242730] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.242763] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.243220] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-3: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.243346] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-5: remote operation failed. Path: <gfid:cd70e63d-a616-47d0-9ab3-f0b8b87d7570> (cd70e63d-a616-47d0-9ab3-f0b8b87d7570). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243378] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-data-client-5: remote operation failed [Transport endpoint is not connected]
[2015-11-24 10:17:46.243390] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-5: remote operation failed. Path: <gfid:506e081c-9d3d-4c7c-8f0a-7d8bb9ef17db> (506e081c-9d3d-4c7c-8f0a-7d8bb9ef17db). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243430] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-5: remote operation failed. Path: <gfid:6cb34603-0f6d-4da1-bc3a-9bfc5a0f493f> (6cb34603-0f6d-4da1-bc3a-9bfc5a0f493f). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243449] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-5: remote operation failed. Path: <gfid:4afb616e-9c48-4b47-9a17-c0ceaeaf0ecb> (4afb616e-9c48-4b47-9a17-c0ceaeaf0ecb). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243465] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-5: remote operation failed. Path: <gfid:7cb6ad2f-afad-4248-b74b-75153682ebee> (7cb6ad2f-afad-4248-b74b-75153682ebee). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243480] W [MSGID: 114061] [client-rpc-fops.c:5882:client3_3_readdir] 0-data-client-5:  (f66aae8e-8c0c-486c-9626-9f7ce494c3eb) remote_fd is -1. EBADFD [File descriptor in bad state]
[2015-11-24 10:17:46.243834] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-3: remote operation failed. Path: <gfid:506e081c-9d3d-4c7c-8f0a-7d8bb9ef17db> (506e081c-9d3d-4c7c-8f0a-7d8bb9ef17db). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243859] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-3: remote operation failed. Path: <gfid:cd70e63d-a616-47d0-9ab3-f0b8b87d7570> (cd70e63d-a616-47d0-9ab3-f0b8b87d7570). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243876] W [MSGID: 114031] [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-data-client-3: remote operation failed. Path: <gfid:526dac6e-1a24-4720-9eda-dfa514c17b09> (526dac6e-1a24-4720-9eda-dfa514c17b09). Key: glusterfs.gfid2path [Transport endpoint is not connected]
[2015-11-24 10:17:46.243892] W [MSGID: 114061] [client-rpc-fops.c:5882:client3_3_readdir] 0-data-client-3:  (38a6318e-4bbc-4c02-adfe-45fec3ab3f6b) remote_fd is -1. EBADFD [File descriptor in bad state]
[2015-11-24 10:17:50.514498] W [socket.c:588:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
[2015-11-24 10:17:50.522044] W [socket.c:588:__socket_rwv] 0-data-client-4: readv on 139.64.206.100:49156 failed (No data available)
[2015-11-24 10:17:50.522076] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-4: disconnected from data-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:50.522095] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-data-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:17:50.526569] W [socket.c:588:__socket_rwv] 0-data-client-0: readv on 139.64.206.100:49152 failed (No data available)
[2015-11-24 10:17:50.526596] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-0: disconnected from data-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:50.526611] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-data-replicate-0: Client-quorum is not met
[2015-11-24 10:17:50.526874] W [socket.c:588:__socket_rwv] 0-data-client-2: readv on 139.64.206.100:49154 failed (No data available)
[2015-11-24 10:17:50.526895] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-2: disconnected from data-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:50.526907] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-data-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-11-24 10:17:56.379967] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.380089] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-1: disconnected from data-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.382493] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-export-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.382620] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-1: disconnected from export-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.385022] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-storage-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.385147] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-3: disconnected from storage-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.387357] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.387478] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-5: disconnected from data-client-5. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.389902] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-iso-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.390025] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-3: disconnected from iso-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.392432] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-export-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.392529] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-export-client-3: disconnected from export-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.394835] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.394931] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-3: disconnected from data-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.397349] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-storage-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.397441] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-storage-client-1: disconnected from storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.399857] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-meta-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.399950] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-meta-client-1: disconnected from meta-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.402361] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvol-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.402454] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-1: disconnected from testvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.404797] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-iso-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.404891] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-iso-client-1: disconnected from iso-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:17:56.407344] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvol-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:17:56.407415] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvol-client-3: disconnected from testvol-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:18:01.441079] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2015-11-24 10:18:01.444027] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:18:01.444091] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-4: disconnected from data-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:18:01.446462] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:18:01.446522] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-0: disconnected from data-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:18:01.448936] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-data-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-11-24 10:18:01.448995] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-data-client-2: disconnected from data-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-11-24 10:18:30.928424] W [socket.c:588:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
[2015-11-24 10:18:32.845406] E [socket.c:2278:socket_connect_finish] 0-data-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.847976] E [socket.c:2278:socket_connect_finish] 0-export-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.850581] E [socket.c:2278:socket_connect_finish] 0-storage-client-3: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.853167] E [socket.c:2278:socket_connect_finish] 0-data-client-5: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.855758] E [socket.c:2278:socket_connect_finish] 0-iso-client-3: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.858355] E [socket.c:2278:socket_connect_finish] 0-export-client-3: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.860935] E [socket.c:2278:socket_connect_finish] 0-data-client-3: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.863512] E [socket.c:2278:socket_connect_finish] 0-storage-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.866134] E [socket.c:2278:socket_connect_finish] 0-meta-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.868728] E [socket.c:2278:socket_connect_finish] 0-testvol-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.871299] E [socket.c:2278:socket_connect_finish] 0-iso-client-1: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:32.873914] E [socket.c:2278:socket_connect_finish] 0-testvol-client-3: connection to 139.64.206.101:24007 failed (Connection refused)
[2015-11-24 10:18:41.968340] E [socket.c:2278:socket_connect_finish] 0-glusterfs: connection to 127.0.0.1:24007 failed (Connection refused)