[2015-07-17 19:54:04.917832] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-0: remote operation failed: No such file or directory
[2015-07-17 19:54:04.917854] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-0: remote operation failed: No such file or directory
[2015-07-17 19:54:04.917882] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-2: remote operation failed: No such file or directory
[2015-07-17 19:54:04.917900] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-1: remote operation failed: No such file or directory
[2015-07-17 19:54:04.917955] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-2: remote operation failed: No such file or directory
[2015-07-17 19:54:04.917987] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-1: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918301] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-0: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918327] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-0: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918594] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-1: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918626] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-1: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918833] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-2: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918840] I [afr-lk-common.c:1092:afr_lock_blocking] 0-systems-replicate-0: unable to lock on even one child
[2015-07-17 19:54:04.918845] I [afr-transaction.c:1245:afr_post_blocking_inodelk_cbk] 0-systems-replicate-0: Blocking inodelks failed.
[2015-07-17 19:54:04.918854] W [nfs3.c:2085:nfs3svc_write_cbk] 0-nfs: 64eef5df: <gfid:7eaaf063-5357-4c57-b560-bb2ae815b9b3>/966 => -1 (No such file or directory)
[2015-07-17 19:54:04.918860] W [nfs3-helpers.c:3453:nfs3_log_write_res] 0-nfs-nfsv3: XID: 64eef5df, WRITE: NFS: 2(No such file or directory), POSIX: 2(No such file or directory), count: 0, STABLE,wverf: 1436968182
[2015-07-17 19:54:04.918906] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-systems-client-2: remote operation failed: No such file or directory
[2015-07-17 19:54:04.918914] I [afr-lk-common.c:1092:afr_lock_blocking] 0-systems-replicate-0: unable to lock on even one child
[2015-07-17 19:54:04.918918] I [afr-transaction.c:1245:afr_post_blocking_inodelk_cbk] 0-systems-replicate-0: Blocking inodelks failed.
[2015-07-17 19:54:04.918924] W [nfs3.c:2085:nfs3svc_write_cbk] 0-nfs: 65eef5df: <gfid:7eaaf063-5357-4c57-b560-bb2ae815b9b3>/966 => -1 (No such file or directory)
[2015-07-17 19:54:04.918929] W [nfs3-helpers.c:3453:nfs3_log_write_res] 0-nfs-nfsv3: XID: 65eef5df, WRITE: NFS: 2(No such file or directory), POSIX: 2(No such file or directory), count: 0, STABLE,wverf: 1436968182

...and repeating...


# gluster volume info

Volume Name: systems
Type: Replicate
Volume ID: b2d72784-4b0e-4f7b-b858-4ec59979a064
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: metal1:/data/gluster/systems
Brick2: metal2:/data/gluster/systems
Brick3: metal3:/data/gluster/systems

Volume Name: docker-repo
Type: Replicate
Volume ID: d762efae-d558-466b-bb4b-8c698ff90569
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: metal1:/data/gluster/docker-repo
Brick2: metal2:/data/gluster/docker-repo
Brick3: metal3:/data/gluster/docker-repo



# gluster volume heal systems info
Brick metal1:/data/gluster/systems/
Number of entries: 0

Brick metal2:/data/gluster/systems/
Number of entries: 0

Brick metal3:/data/gluster/systems/
Number of entries: 0


# gluster peer status
Number of Peers: 2

Hostname: metal2
Uuid: eb794bf4-7cd7-42a6-bf38-a75fc5b6a776
State: Peer in Cluster (Connected)

Hostname: metal3
Uuid: 5b69e353-f95c-478a-ab25-3f77d36ba8b8
State: Peer in Cluster (Connected)



# gluster --version
glusterfs 3.5.2 built on Aug 20 2014 13:10:49
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.