[2011-06-30 16:42:57.262489] W [io-stats.c:1644:init] 0-storage_clu1: dangling volume. check volfile
[2011-06-30 16:42:57.262529] W [dict.c:1205:data_to_str] 0-dict: @data=(nil)
[2011-06-30 16:42:57.262541] W [dict.c:1205:data_to_str] 0-dict: @data=(nil)
Given volfile:
+------------------------------------------------------------------------------+
  1: volume storage_clu1-client-0
  2:     type protocol/client
  3:     option remote-host snode01
  4:     option remote-subvolume /cluster/0
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume storage_clu1-client-1
  9:     type protocol/client
 10:     option remote-host snode02
 11:     option remote-subvolume /cluster/0
 12:     option transport-type tcp
 13: end-volume
 14:
 15: volume storage_clu1-client-2
 16:     type protocol/client
 17:     option remote-host snode01
 18:     option remote-subvolume /cluster/1
 19:     option transport-type tcp
 20: end-volume
 21:
 22: volume storage_clu1-client-3
 23:     type protocol/client
 24:     option remote-host snode02
 25:     option remote-subvolume /cluster/1
 26:     option transport-type tcp
 27: end-volume
 28:
 29: volume storage_clu1-client-4
 30:     type protocol/client
 31:     option remote-host snode01
 32:     option remote-subvolume /cluster/2
 33:     option transport-type tcp
 34: end-volume
 35:
 36: volume storage_clu1-client-5
 37:     type protocol/client
 38:     option remote-host snode02
 39:     option remote-subvolume /cluster/2
 40:     option transport-type tcp
 41: end-volume
 42:
 43: volume storage_clu1-client-6
 44:     type protocol/client
 45:     option remote-host snode01
 46:     option remote-subvolume /cluster/3
 47:     option transport-type tcp
 48: end-volume
 49:
 50: volume storage_clu1-client-7
 51:     type protocol/client
 52:     option remote-host snode02
 53:     option remote-subvolume /cluster/3
 54:     option transport-type tcp
 55: end-volume
 56:
 57: volume storage_clu1-replicate-0
 58:     type cluster/replicate
 59:     subvolumes storage_clu1-client-0 storage_clu1-client-1
 60: end-volume
 61:
 62: volume storage_clu1-replicate-1
 63:     type cluster/replicate
 64:     subvolumes storage_clu1-client-2 storage_clu1-client-3
 65: end-volume
 66:
 67: volume storage_clu1-replicate-2
 68:     type cluster/replicate
 69:     subvolumes storage_clu1-client-4 storage_clu1-client-5
 70: end-volume
 71:
 72: volume storage_clu1-replicate-3
 73:     type cluster/replicate
 74:     subvolumes storage_clu1-client-6 storage_clu1-client-7
 75: end-volume
 76:
 77: volume storage_clu1-dht
 78:     type cluster/distribute
 79:     subvolumes storage_clu1-replicate-0 storage_clu1-replicate-1 storage_clu1-replicate-2 storage_clu1-replicate-3
 80: end-volume
 81:
 82: volume storage_clu1-write-behind
 83:     type performance/write-behind
 84:     subvolumes storage_clu1-dht
 85: end-volume
 86:
 87: volume storage_clu1-read-ahead
 88:     type performance/read-ahead
 89:     subvolumes storage_clu1-write-behind
 90: end-volume
 91:
 92: volume storage_clu1-io-cache
 93:     type performance/io-cache
 94:     option cache-size 512MB
 95:     subvolumes storage_clu1-read-ahead
 96: end-volume
 97:
 98: volume storage_clu1-quick-read
 99:     type performance/quick-read
100:     option cache-size 512MB
101:     subvolumes storage_clu1-io-cache
102: end-volume
103:
104: volume storage_clu1-stat-prefetch
105:     type performance/stat-prefetch
106:     subvolumes storage_clu1-quick-read
107: end-volume
108:
109: volume storage_clu1
110:     type debug/io-stats
111:     subvolumes storage_clu1-stat-prefetch
112: end-volume

+------------------------------------------------------------------------------+
[2011-06-30 16:43:01.157953] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158011] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-5: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158049] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158341] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-7: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158428] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-5: Connected to 10.10.0.2:24011, attached to remote volume '/cluster/2'.
[2011-06-30 16:43:01.158444] I [afr-common.c:2644:afr_notify] 0-storage_clu1-replicate-2: Subvolume 'storage_clu1-client-5' came back up; going online.
[2011-06-30 16:43:01.158466] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-3: Connected to 10.10.0.2:24010, attached to remote volume '/cluster/1'.
[2011-06-30 16:43:01.158497] I [afr-common.c:2644:afr_notify] 0-storage_clu1-replicate-1: Subvolume 'storage_clu1-client-3' came back up; going online.
[2011-06-30 16:43:01.158525] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-1: Connected to 10.10.0.2:24009, attached to remote volume '/cluster/0'.
[2011-06-30 16:43:01.158535] I [afr-common.c:2644:afr_notify] 0-storage_clu1-replicate-0: Subvolume 'storage_clu1-client-1' came back up; going online.
[2011-06-30 16:43:01.158575] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158612] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-6: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158740] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-7: Connected to 10.10.0.2:24012, attached to remote volume '/cluster/3'.
[2011-06-30 16:43:01.158773] I [afr-common.c:2644:afr_notify] 0-storage_clu1-replicate-3: Subvolume 'storage_clu1-client-7' came back up; going online.
[2011-06-30 16:43:01.158827] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-4: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:01.158863] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-2: Connected to 10.10.0.1:24010, attached to remote volume '/cluster/1'.
[2011-06-30 16:43:01.158929] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-6: Connected to 10.10.0.1:24012, attached to remote volume '/cluster/3'.
[2011-06-30 16:43:01.159332] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-4: Connected to 10.10.0.1:24011, attached to remote volume '/cluster/2'.
[2011-06-30 16:43:04.159530] I [client-handshake.c:1016:select_server_supported_programs] 0-storage_clu1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-06-30 16:43:04.160038] I [client-handshake.c:852:client_setvolume_cbk] 0-storage_clu1-client-0: Connected to 10.10.0.1:24009, attached to remote volume '/cluster/0'.
[2011-06-30 16:43:04.162230] I [fuse-bridge.c:3316:fuse_graph_setup] 0-fuse: switched graph to 0
[2011-06-30 16:43:04.162368] I [fuse-bridge.c:2897:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2011-06-30 16:43:04.162997] I [afr-common.c:893:afr_fresh_lookup_cbk] 0-storage_clu1-replicate-0: added root inode
[2011-06-30 16:43:04.163661] I [afr-common.c:893:afr_fresh_lookup_cbk] 0-storage_clu1-replicate-1: added root inode
[2011-06-30 16:43:04.163720] I [afr-common.c:893:afr_fresh_lookup_cbk] 0-storage_clu1-replicate-2: added root inode
[2011-06-30 16:43:04.163741] I [afr-common.c:893:afr_fresh_lookup_cbk] 0-storage_clu1-replicate-3: added root inode
[2011-06-30 16:43:06.993090] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] 0-storage_clu1-replicate-3:  entry self-heal triggered. path: /, reason: checksums of directory differ, forced merge option set
[2011-06-30 16:43:06.993157] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] 0-storage_clu1-replicate-0:  entry self-heal triggered. path: /, reason: checksums of directory differ, forced merge option set
[2011-06-30 16:43:06.993448] E [afr-self-heal-entry.c:2274:afr_sh_post_nonblocking_entry_cbk] 0-storage_clu1-replicate-3: Non Blocking entrylks failed for /.
[2011-06-30 16:43:06.993485] E [afr-common.c:120:afr_set_split_brain] 0-storage_clu1-replicate-3: invalid argument: inode
[2011-06-30 16:43:06.993494] E [afr-self-heal-common.c:1533:afr_self_heal_completion_cbk] 0-storage_clu1-replicate-3: background  entry self-heal failed on /
[2011-06-30 16:43:06.993515] E [afr-self-heal-entry.c:2274:afr_sh_post_nonblocking_entry_cbk] 0-storage_clu1-replicate-0: Non Blocking entrylks failed for /.
[2011-06-30 16:43:06.993525] E [afr-common.c:120:afr_set_split_brain] 0-storage_clu1-replicate-0: invalid argument: inode
[2011-06-30 16:43:06.993532] E [afr-self-heal-common.c:1533:afr_self_heal_completion_cbk] 0-storage_clu1-replicate-0: background  entry self-heal failed on /