[2017-07-11 14:56:31.951080] I [glusterfsd.c:1959:main] 0-glusterfs: Started running glusterfs version 3.5.2 (glusterfs --volfile-server=gfs-01-01 --volfile-id=LM-HA-01 /LM-HA-01)
[2017-07-11 14:56:31.962365] I [socket.c:3578:socket_init] 0-glusterfs: SSL support is NOT enabled
[2017-07-11 14:56:31.962406] I [socket.c:3593:socket_init] 0-glusterfs: using system polling thread
[2017-07-11 14:56:31.973968] I [dht-shared.c:311:dht_init_regex] 0-LM-HA-01-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2017-07-11 14:56:31.975346] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-LM-HA-01-client-2: setting frame-timeout to 120
[2017-07-11 14:56:31.975410] I [socket.c:3578:socket_init] 0-LM-HA-01-client-2: SSL support is ENABLED
[2017-07-11 14:56:31.975433] I [socket.c:3593:socket_init] 0-LM-HA-01-client-2: using private polling thread
[2017-07-11 14:56:31.977281] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-LM-HA-01-client-1: setting frame-timeout to 120
[2017-07-11 14:56:31.977334] I [socket.c:3578:socket_init] 0-LM-HA-01-client-1: SSL support is ENABLED
[2017-07-11 14:56:31.977349] I [socket.c:3593:socket_init] 0-LM-HA-01-client-1: using private polling thread
[2017-07-11 14:56:31.977815] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-LM-HA-01-client-0: setting frame-timeout to 120
[2017-07-11 14:56:31.977852] I [socket.c:3578:socket_init] 0-LM-HA-01-client-0: SSL support is ENABLED
[2017-07-11 14:56:31.977866] I [socket.c:3593:socket_init] 0-LM-HA-01-client-0: using private polling thread
[2017-07-11 14:56:31.978161] I [client.c:2294:notify] 0-LM-HA-01-client-0: parent translators are ready, attempting connect on transport
[2017-07-11 14:56:31.982733] I [client.c:2294:notify] 0-LM-HA-01-client-1: parent translators are ready, attempting connect on transport
[2017-07-11 14:56:31.987163] I [client.c:2294:notify] 0-LM-HA-01-client-2: parent translators are ready, attempting connect on transport
[2017-07-11 14:56:31.991623] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-LM-HA-01-client-0: changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume LM-HA-01-client-0
  2:     type protocol/client
  3:     option remote-host gfs-01-01
  4:     option remote-subvolume /mnt/zfspool/LM-HA-01
  5:     option transport-type socket
  6:     option transport.socket.ssl-enabled on
  7:     option frame-timeout 120
  8:     option ping-timeout 3
  9:     option send-gids true
 10: end-volume
 11:
 12: volume LM-HA-01-client-1
 13:     type protocol/client
 14:     option remote-host gfs-01-03
 15:     option remote-subvolume /mnt/zfspool/LM-HA-01
 16:     option transport-type socket
 17:     option transport.socket.ssl-enabled on
 18:     option frame-timeout 120
 19:     option ping-timeout 3
 20:     option send-gids true
 21: end-volume
 22:
 23: volume LM-HA-01-client-2
 24:     type protocol/client
 25:     option remote-host gfs-01-02
 26:     option remote-subvolume /mnt/zfspool/LM-HA-01
 27:     option transport-type socket
 28:     option transport.socket.ssl-enabled on
 29:     option frame-timeout 120
 30:     option ping-timeout 3
 31:     option send-gids true
 32: end-volume
 33:
 34: volume LM-HA-01-replicate-0
 35:     type cluster/replicate
 36:     subvolumes LM-HA-01-client-0 LM-HA-01-client-1 LM-HA-01-client-2
 37: end-volume
 38:
 39: volume LM-HA-01-dht
 40:     type cluster/distribute
 41:     subvolumes LM-HA-01-replicate-0
 42: end-volume
 43:
 44: volume LM-HA-01-write-behind
 45:     type performance/write-behind
 46:     subvolumes LM-HA-01-dht
 47: end-volume
 48:
 49: volume LM-HA-01-read-ahead
 50:     type performance/read-ahead
 51:     subvolumes LM-HA-01-write-behind
 52: end-volume
 53:
 54: volume LM-HA-01-io-cache
 55:     type performance/io-cache
 56:     subvolumes LM-HA-01-read-ahead
 57: end-volume
 58:
 59: volume LM-HA-01-quick-read
 60:     type performance/quick-read
 61:     subvolumes LM-HA-01-io-cache
 62: end-volume
 63:
 64: volume LM-HA-01-open-behind
 65:     type performance/open-behind
 66:     subvolumes LM-HA-01-quick-read
 67: end-volume
 68:
 69: volume LM-HA-01-md-cache
 70:     type performance/md-cache
 71:     subvolumes LM-HA-01-open-behind
 72: end-volume
 73:
 74: volume LM-HA-01
 75:     type debug/io-stats
 76:     option latency-measurement on
 77:     option count-fop-hits on
 78:     subvolumes LM-HA-01-md-cache
 79: end-volume
 80:
+------------------------------------------------------------------------------+
[2017-07-11 14:56:31.996387] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-LM-HA-01-client-1: changing port to 49152 (from 0)
[2017-07-11 14:56:32.000576] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-LM-HA-01-client-2: changing port to 49152 (from 0)
[2017-07-11 14:56:32.047599] I [socket.c:292:ssl_setup_connection] 0-LM-HA-01-client-0: peer CN = LMB
[2017-07-11 14:56:32.052152] I [client-handshake.c:1677:select_server_supported_programs] 0-LM-HA-01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-07-11 14:56:32.053923] I [socket.c:292:ssl_setup_connection] 0-LM-HA-01-client-1: peer CN = LMB
[2017-07-11 14:56:32.056741] I [client-handshake.c:1462:client_setvolume_cbk] 0-LM-HA-01-client-0: Connected to xxx.yyy..104.144:49152, attached to remote volume '/mnt/zfspool/LM-HA-01'.
[2017-07-11 14:56:32.056765] I [client-handshake.c:1474:client_setvolume_cbk] 0-LM-HA-01-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-07-11 14:56:32.056875] I [afr-common.c:4131:afr_notify] 0-LM-HA-01-replicate-0: Subvolume 'LM-HA-01-client-0' came back up; going online.
[2017-07-11 14:56:32.058414] I [client-handshake.c:1677:select_server_supported_programs] 0-LM-HA-01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-07-11 14:56:32.058664] I [socket.c:292:ssl_setup_connection] 0-LM-HA-01-client-2: peer CN = LMB
[2017-07-11 14:56:32.061128] I [client-handshake.c:450:client_set_lk_version_cbk] 0-LM-HA-01-client-0: Server lk version = 1
[2017-07-11 14:56:32.063178] I [client-handshake.c:1462:client_setvolume_cbk] 0-LM-HA-01-client-1: Connected to xxx.yyy..104.146:49152, attached to remote volume '/mnt/zfspool/LM-HA-01'.
[2017-07-11 14:56:32.063198] I [client-handshake.c:1474:client_setvolume_cbk] 0-LM-HA-01-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-07-11 14:56:32.063335] I [client-handshake.c:1677:select_server_supported_programs] 0-LM-HA-01-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-07-11 14:56:32.067582] I [client-handshake.c:450:client_set_lk_version_cbk] 0-LM-HA-01-client-1: Server lk version = 1
[2017-07-11 14:56:32.067947] I [client-handshake.c:1462:client_setvolume_cbk] 0-LM-HA-01-client-2: Connected to xxx.yyy..104.145:49152, attached to remote volume '/mnt/zfspool/LM-HA-01'.
[2017-07-11 14:56:32.067966] I [client-handshake.c:1474:client_setvolume_cbk] 0-LM-HA-01-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-07-11 14:56:32.071471] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
[2017-07-11 14:56:32.071605] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.14
[2017-07-11 14:56:32.072431] I [client-handshake.c:450:client_set_lk_version_cbk] 0-LM-HA-01-client-2: Server lk version = 1
[2017-07-11 15:02:50.029638] W [socket.c:523:__socket_rwv] 0-LM-HA-01-client-2: readv on xxx.yyy..104.145:49152 failed (No data available)
[2017-07-11 15:02:50.029686] E [socket.c:2384:socket_poller] 0-LM-HA-01-client-2: poll error on socket
[2017-07-11 15:02:50.029737] I [client.c:2229:client_rpc_notify] 0-LM-HA-01-client-2: disconnected from xxx.yyy..104.145:49152. Client process will keep trying to connect to glusterd until brick's port is available
[2017-07-11 15:02:59.263651] W [socket.c:523:__socket_rwv] 2-LM-HA-01-client-2: readv on xxx.yyy..104.145:49152 failed (No data available)
[2017-07-11 15:02:59.263695] E [socket.c:2384:socket_poller] 2-LM-HA-01-client-2: poll error on socket
[2017-07-11 15:02:59.263745] I [client.c:2229:client_rpc_notify] 2-LM-HA-01-client-2: disconnected from xxx.yyy..104.145:49152. Client process will keep trying to connect to glusterd until brick's port is available
[2017-07-11 15:04:03.793814] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:04:13.114598] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:05:09.793580] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:05:19.115569] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:06:15.793566] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:06:25.115562] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:07:21.793597] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:07:31.115566] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:08:27.793600] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:08:37.115560] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:09:33.793596] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:09:43.115603] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:10:39.793597] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:10:49.115567] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:11:45.793585] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:11:55.115564] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:12:51.793587] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:13:01.115571] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:13:57.793565] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:14:07.115631] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:15:03.793821] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:15:13.116587] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:16:09.793582] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:16:19.116608] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:17:15.793598] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:17:25.116578] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:17:59.127627] W [socket.c:523:__socket_rwv] 0-LM-HA-01-client-2: writev on xxx.yyy..104.145:49152 failed (No data available)
[2017-07-11 15:17:59.127768] W [socket.c:523:__socket_rwv] 0-LM-HA-01-client-2: readv on xxx.yyy..104.145:49152 failed (No data available)
[2017-07-11 15:17:59.127786] E [socket.c:2384:socket_poller] 0-LM-HA-01-client-2: poll error on socket
[2017-07-11 15:17:59.128124] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48) [0x7f591249ec38] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb7) [0x7f591249cf07] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f591249ce2e]))) 0-LM-HA-01-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2017-07-11 15:02:13.943077 (xid=0x1ffb)
[2017-07-11 15:17:59.128140] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:17:59.128172] W [afr-transaction.c:1700:afr_changelog_fsync_cbk] 0-LM-HA-01-replicate-0: fsync(d6679344-7820-4c8a-b92a-309fe657afc0) failed on subvolume LM-HA-01-client-2. Transaction was WRITE
[2017-07-11 15:18:21.793567] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:18:31.116586] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:19:02.128589] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:19:02.128653] I [socket.c:3077:socket_submit_request] 0-LM-HA-01-client-2: not connected (priv->connected = -1)
[2017-07-11 15:19:02.128669] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2431 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:19:02.128689] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:19:27.793566] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:19:38.154567] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:20:05.128580] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:20:05.128641] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2432 Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:20:05.128666] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:20:05.128737] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48) [0x7f591249ec38] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb7) [0x7f591249cf07] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f591249ce2e]))) 0-LM-HA-01-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2017-07-11 15:02:13.943273 (xid=0x1ffd)
[2017-07-11 15:20:05.128750] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:20:05.128765] W [afr-transaction.c:1700:afr_changelog_fsync_cbk] 0-LM-HA-01-replicate-0: fsync(d6679344-7820-4c8a-b92a-309fe657afc0) failed on subvolume LM-HA-01-client-2. Transaction was WRITE
[2017-07-11 15:20:33.793566] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:20:44.154590] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:21:08.128606] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:21:08.128678] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2433 Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:21:08.128701] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:21:08.128746] W [client-rpc-fops.c:4201:client3_3_writev] 0-LM-HA-01-client-2: failed to send the fop
[2017-07-11 15:21:39.793582] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:21:50.154564] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:22:11.128613] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:22:11.128688] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2434 Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:22:11.128709] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:22:45.793566] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:22:56.154603] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:23:14.128574] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:23:14.128649] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2435 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:23:14.128674] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2017-07-11 15:23:51.793576] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:24:02.154605] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:24:17.128604] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:24:17.351926] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2436 Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:24:17.351960] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:24:17.352032] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48) [0x7f591249ec38] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb7) [0x7f591249cf07] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f591249ce2e]))) 0-LM-HA-01-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2017-07-11 15:02:13.943360 (xid=0x1ffe)
[2017-07-11 15:24:17.352043] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:24:17.352060] W [afr-transaction.c:1700:afr_changelog_fsync_cbk] 0-LM-HA-01-replicate-0: fsync(d6679344-7820-4c8a-b92a-309fe657afc0) failed on subvolume LM-HA-01-client-2. Transaction was WRITE
[2017-07-11 15:24:57.793827] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:25:08.154574] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:25:20.351596] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:25:20.351664] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2437 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:25:20.351687] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2017-07-11 15:26:03.793607] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:26:14.154613] E [socket.c:2837:socket_connect] 2-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:26:23.351592] E [socket.c:2837:socket_connect] 0-LM-HA-01-client-2: connection attempt on xxx.yyy..104.145:49152 failed, (Connection timed out)
[2017-07-11 15:26:23.351650] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-LM-HA-01-client-2: failed to submit rpc-request (XID: 0x2438 Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (LM-HA-01-client-2)
[2017-07-11 15:26:23.351671] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:26:23.351738] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48) [0x7f591249ec38] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb7) [0x7f591249cf07] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f591249ce2e]))) 0-LM-HA-01-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2017-07-11 15:02:13.943449 (xid=0x1fff)
[2017-07-11 15:26:23.351750] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 0-LM-HA-01-client-2: remote operation failed: Transport endpoint is not connected
[2017-07-11 15:26:23.351764] W [afr-transaction.c:1700:afr_changelog_fsync_cbk] 0-LM-HA-01-replicate-0: fsync(d6679344-7820-4c8a-b92a-309fe657afc0) failed on subvolume LM-HA-01-client-2. Transaction was WRITE