Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Drbd after fencing with rhcs_fence with debug

By: a guest on Jan 25th, 2012  |  syntax: Bash  |  size: 34.20 KB  |  views: 42  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. Log in node 1
  2.  
  3. Jan 25 15:02:43 wsguardian1 kernel: igb: eth4 NIC Link is Down
  4. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: PingAck did not arrive in time.
  5. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  6. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: asender terminated
  7. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: Terminating asender thread
  8. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: short read expecting header on sock: r=-512
  9. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: Creating new current UUID
  10. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: Connection closed
  11. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  12. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  13. Jan 25 15:02:52 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  14. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  15. Jan 25 15:02:52 wsguardian1 rhcs_fence: 128; DEBUG: It is not!
  16. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/sbin/fence_node]
  17. Jan 25 15:02:52 wsguardian1 rhcs_fence: 136; DEBUG: Found!
  18. Jan 25 15:02:52 wsguardian1 rhcs_fence: 146; DEBUG: Set 'path::fence_node' to: [/sbin/fence_node]
  19. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/usr/sbin/fence_node]
  20. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  21. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/bin/fence_node]
  22. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  23. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/usr/bin/fence_node]
  24. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  25. Jan 25 15:02:52 wsguardian1 rhcs_fence: 161; DEBUG: Checking again if: [fence_node] is at: [/sbin/fence_node].
  26. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  27. Jan 25 15:02:52 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  28. Jan 25 15:02:52 wsguardian1 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  29. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_db]
  30. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [0]
  31. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian2]
  32. Jan 25 15:02:52 wsguardian1 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  33. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Version: 6.2.0
  34. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Config Version: 2
  35. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Name: WSGClust
  36. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Id: 21470
  37. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Member: Yes
  38. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Generation: 328
  39. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Membership state: Cluster-Member
  40. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Nodes: 2
  41. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Expected votes: 1
  42. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Total votes: 2
  43. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Node votes: 1
  44. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Quorum: 1  
  45. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Active subsystems: 7
  46. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Flags: 2node Dirty
  47. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Ports Bound: 0 11  
  48. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Node name: wsguardian1
  49. Jan 25 15:02:52 wsguardian1 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [0]
  50. Jan 25 15:02:52 wsguardian1 rhcs_fence: 471; DEBUG: I am: [wsguardian1]
  51. Jan 25 15:02:52 wsguardian1 rhcs_fence: 403; DEBUG: Checking the state of resource with minor number: [0]
  52. Jan 25 15:02:52 wsguardian1 rhcs_fence: 409; DEBUG: minor: [0]
  53. Jan 25 15:02:52 wsguardian1 rhcs_fence: 412; DEBUG: shell call: [</proc/drbd]
  54. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output: version: 8.3.8 (api:88/proto:86-94)
  55. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output: GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09
  56. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:  0: cs:NetworkFailure ro:Primary/Unknown ds:UpToDate/DUnknown C s----
  57. Jan 25 15:02:52 wsguardian1 rhcs_fence: 423; DEBUG: read state of minor: [0] as: [UpToDate]
  58. Jan 25 15:02:52 wsguardian1 rhcs_fence: 425; DEBUG: sys::local_res_uptodate: [1]
  59. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:     ns:276 nr:500 dw:776 dr:4656 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  60. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:  1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
  61. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:     ns:68 nr:76 dw:144 dr:4280 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  62. Jan 25 15:02:52 wsguardian1 rhcs_fence: 430; DEBUG: Attempt to collect UpToDate information device with minor: [0] exited with: [0]
  63. Jan 25 15:02:52 wsguardian1 rhcs_fence: 432; DEBUG: UpToDate: [1]
  64. Jan 25 15:02:52 wsguardian1 rhcs_fence: 367; DEBUG: shell call: [/usr/sbin/cman_tool -a -F id,name,type,addr nodes]
  65. Jan 25 15:02:52 wsguardian1 rhcs_fence: 374; DEBUG: id: [1], name: [wsguardian1], member: [M], address: [192.168.253.1]
  66. Jan 25 15:02:52 wsguardian1 rhcs_fence: 380; DEBUG: output: 1 wsguardian1 M 192.168.253.1
  67. Jan 25 15:02:52 wsguardian1 rhcs_fence: 374; DEBUG: id: [2], name: [wsguardian2], member: [M], address: [192.168.253.2]
  68. Jan 25 15:02:52 wsguardian1 rhcs_fence: 380; DEBUG: output: 2 wsguardian2 M 192.168.253.2
  69. Jan 25 15:02:52 wsguardian1 rhcs_fence: 384; DEBUG: Attempt to gather cluster member information exited with: [0]
  70. Jan 25 15:02:52 wsguardian1 rhcs_fence: 343; I have identified my target: [wsguardian2]
  71. Jan 25 15:02:52 wsguardian1 rhcs_fence: 191; DEBUG: I am: [wsguardian1] and my id is: [1]
  72. Jan 25 15:02:52 wsguardian1 rhcs_fence: 223; I am the first node, so I won't delay.
  73. Jan 25 15:02:52 wsguardian1 rhcs_fence: 301; Target node: [wsguardian2] is a cluster member, attempting to eject.
  74. Jan 25 15:02:52 wsguardian1 rhcs_fence: 304; DEBUG: shell call: [/usr/sbin/cman_tool kill -f wsguardian2]
  75. Jan 25 15:02:52 wsguardian1 rhcs_fence: 313; DEBUG: Attempt to force-remove node: [wsguardian2] exited with: [256]
  76. Jan 25 15:02:52 wsguardian1 rhcs_fence: 243; Fencing target: [wsguardian2]...
  77. Jan 25 15:02:52 wsguardian1 rhcs_fence: 247; DEBUG: shell call: [/sbin/fence_node -v wsguardian2]
  78. Jan 25 15:02:52 wsguardian1 rhcs_fence: 253; DEBUG: /sbin/fence_node: invalid option -- v
  79. Jan 25 15:02:52 wsguardian1 rhcs_fence: 260; 'fence_node wsguardian2' appears to have failed!
  80. Jan 25 15:02:52 wsguardian1 rhcs_fence: 261; Read: [/sbin/fence_node: invalid option -- v]
  81. Jan 25 15:02:52 wsguardian1 rhcs_fence: 253; DEBUG: Please use '-h' for usage.
  82. Jan 25 15:02:52 wsguardian1 rhcs_fence: 260; 'fence_node wsguardian2' appears to have failed!
  83. Jan 25 15:02:52 wsguardian1 rhcs_fence: 261; Read: [Please use '-h' for usage.]
  84. Jan 25 15:02:52 wsguardian1 rhcs_fence: 266; DEBUG: Attempt to fence node: [wsguardian2] exited with: [256]
  85. Jan 25 15:02:52 wsguardian1 rhcs_fence: 275; Fencing of: [wsguardian2] succeeded!
  86. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 7 (0x700)
  87. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: fence-peer helper returned 7 (peer was stonithed)
  88. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: pdsk( DUnknown -> Outdated )
  89. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: susp( 1 -> 0 )
  90. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: conn( NetworkFailure -> Unconnected )
  91. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: receiver terminated
  92. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: Restarting receiver thread
  93. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: receiver (re)started
  94. Jan 25 15:02:52 wsguardian1 kernel: block drbd0: conn( Unconnected -> WFConnection )
  95. Jan 25 15:02:52 wsguardian1 openais[12622]: [TOTEM] The token was lost in the OPERATIONAL state.
  96. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: PingAck did not arrive in time.
  97. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  98. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: asender terminated
  99. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: Terminating asender thread
  100. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: short read expecting header on sock: r=-512
  101. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: Creating new current UUID
  102. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: Connection closed
  103. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  104. Jan 25 15:02:52 wsguardian1 openais[12622]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
  105. Jan 25 15:02:52 wsguardian1 openais[12622]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
  106. Jan 25 15:02:52 wsguardian1 openais[12622]: [TOTEM] entering GATHER state from 2.
  107. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  108. Jan 25 15:02:52 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  109. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  110. Jan 25 15:02:52 wsguardian1 rhcs_fence: 128; DEBUG: It is not!
  111. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/sbin/fence_node]
  112. Jan 25 15:02:52 wsguardian1 rhcs_fence: 136; DEBUG: Found!
  113. Jan 25 15:02:52 wsguardian1 rhcs_fence: 146; DEBUG: Set 'path::fence_node' to: [/sbin/fence_node]
  114. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/usr/sbin/fence_node]
  115. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  116. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/bin/fence_node]
  117. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  118. Jan 25 15:02:52 wsguardian1 rhcs_fence: 133; DEBUG: Checking: [/usr/bin/fence_node]
  119. Jan 25 15:02:52 wsguardian1 rhcs_fence: 150; DEBUG: Not found!
  120. Jan 25 15:02:52 wsguardian1 rhcs_fence: 161; DEBUG: Checking again if: [fence_node] is at: [/sbin/fence_node].
  121. Jan 25 15:02:52 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  122. Jan 25 15:02:52 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  123. Jan 25 15:02:52 wsguardian1 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  124. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_config]
  125. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [1]
  126. Jan 25 15:02:52 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian2]
  127. Jan 25 15:02:52 wsguardian1 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  128. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Version: 6.2.0
  129. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Config Version: 2
  130. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Name: WSGClust
  131. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Id: 21470
  132. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Member: Yes
  133. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Cluster Generation: 328
  134. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Membership state: Cluster-Member
  135. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Nodes: 2
  136. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Expected votes: 1
  137. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Total votes: 2
  138. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Node votes: 1
  139. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Quorum: 1  
  140. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Active subsystems: 7
  141. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Flags: 2node Dirty
  142. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Ports Bound: 0 11  
  143. Jan 25 15:02:52 wsguardian1 rhcs_fence: 460; DEBUG: output: Node name: wsguardian1
  144. Jan 25 15:02:52 wsguardian1 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [0]
  145. Jan 25 15:02:52 wsguardian1 rhcs_fence: 471; DEBUG: I am: [wsguardian1]
  146. Jan 25 15:02:52 wsguardian1 rhcs_fence: 403; DEBUG: Checking the state of resource with minor number: [1]
  147. Jan 25 15:02:52 wsguardian1 rhcs_fence: 409; DEBUG: minor: [1]
  148. Jan 25 15:02:52 wsguardian1 rhcs_fence: 412; DEBUG: shell call: [</proc/drbd]
  149. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output: version: 8.3.8 (api:88/proto:86-94)
  150. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output: GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09
  151. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:  0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/Outdated C r----
  152. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:     ns:276 nr:500 dw:776 dr:4656 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  153. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:  1: cs:NetworkFailure ro:Primary/Unknown ds:UpToDate/DUnknown C s----
  154. Jan 25 15:02:52 wsguardian1 rhcs_fence: 423; DEBUG: read state of minor: [1] as: [UpToDate]
  155. Jan 25 15:02:52 wsguardian1 rhcs_fence: 425; DEBUG: sys::local_res_uptodate: [1]
  156. Jan 25 15:02:52 wsguardian1 rhcs_fence: 418; DEBUG: output:     ns:68 nr:76 dw:144 dr:4280 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  157. Jan 25 15:02:52 wsguardian1 rhcs_fence: 430; DEBUG: Attempt to collect UpToDate information device with minor: [1] exited with: [0]
  158. Jan 25 15:02:52 wsguardian1 rhcs_fence: 432; DEBUG: UpToDate: [1]
  159. Jan 25 15:02:52 wsguardian1 rhcs_fence: 367; DEBUG: shell call: [/usr/sbin/cman_tool -a -F id,name,type,addr nodes]
  160. Jan 25 15:02:52 wsguardian1 rhcs_fence: 374; DEBUG: id: [1], name: [wsguardian1], member: [M], address: [192.168.253.1]
  161. Jan 25 15:02:52 wsguardian1 rhcs_fence: 380; DEBUG: output: 1 wsguardian1 M 192.168.253.1
  162. Jan 25 15:02:52 wsguardian1 rhcs_fence: 374; DEBUG: id: [2], name: [wsguardian2], member: [M], address: [192.168.253.2]
  163. Jan 25 15:02:52 wsguardian1 rhcs_fence: 380; DEBUG: output: 2 wsguardian2 M 192.168.253.2
  164. Jan 25 15:02:52 wsguardian1 rhcs_fence: 384; DEBUG: Attempt to gather cluster member information exited with: [0]
  165. Jan 25 15:02:52 wsguardian1 rhcs_fence: 343; I have identified my target: [wsguardian2]
  166. Jan 25 15:02:52 wsguardian1 rhcs_fence: 191; DEBUG: I am: [wsguardian1] and my id is: [1]
  167. Jan 25 15:02:52 wsguardian1 rhcs_fence: 223; I am the first node, so I won't delay.
  168. Jan 25 15:02:52 wsguardian1 rhcs_fence: 301; Target node: [wsguardian2] is a cluster member, attempting to eject.
  169. Jan 25 15:02:52 wsguardian1 rhcs_fence: 304; DEBUG: shell call: [/usr/sbin/cman_tool kill -f wsguardian2]
  170. Jan 25 15:02:52 wsguardian1 rhcs_fence: 313; DEBUG: Attempt to force-remove node: [wsguardian2] exited with: [256]
  171. Jan 25 15:02:52 wsguardian1 rhcs_fence: 243; Fencing target: [wsguardian2]...
  172. Jan 25 15:02:52 wsguardian1 rhcs_fence: 247; DEBUG: shell call: [/sbin/fence_node -v wsguardian2]
  173. Jan 25 15:02:52 wsguardian1 rhcs_fence: 253; DEBUG: /sbin/fence_node: invalid option -- v
  174. Jan 25 15:02:52 wsguardian1 rhcs_fence: 260; 'fence_node wsguardian2' appears to have failed!
  175. Jan 25 15:02:52 wsguardian1 rhcs_fence: 261; Read: [/sbin/fence_node: invalid option -- v]
  176. Jan 25 15:02:52 wsguardian1 rhcs_fence: 253; DEBUG: Please use '-h' for usage.
  177. Jan 25 15:02:52 wsguardian1 rhcs_fence: 260; 'fence_node wsguardian2' appears to have failed!
  178. Jan 25 15:02:52 wsguardian1 rhcs_fence: 261; Read: [Please use '-h' for usage.]
  179. Jan 25 15:02:52 wsguardian1 rhcs_fence: 266; DEBUG: Attempt to fence node: [wsguardian2] exited with: [256]
  180. Jan 25 15:02:52 wsguardian1 rhcs_fence: 275; Fencing of: [wsguardian2] succeeded!
  181. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 7 (0x700)
  182. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: fence-peer helper returned 7 (peer was stonithed)
  183. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: pdsk( DUnknown -> Outdated )
  184. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: susp( 1 -> 0 )
  185. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: conn( NetworkFailure -> Unconnected )
  186. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: receiver terminated
  187. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: Restarting receiver thread
  188. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: receiver (re)started
  189. Jan 25 15:02:52 wsguardian1 kernel: block drbd1: conn( Unconnected -> WFConnection )
  190. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] entering GATHER state from 0.
  191. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] Creating commit token because I am the rep.
  192. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] Storing new sequence id for ring 14c
  193. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] entering COMMIT state.
  194. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] entering RECOVERY state.
  195. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] position [0] member 192.168.253.1:
  196. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] previous ring seq 328 rep 192.168.253.1
  197. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] aru b3 high delivered b3 received flag 1
  198. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] Did not need to originate any messages in recovery.
  199. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] Sending initial ORF token
  200. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] CLM CONFIGURATION CHANGE
  201. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] New Configuration:
  202. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ]     r(0) ip(192.168.253.1)  
  203. Jan 25 15:02:54 wsguardian1 fenced[12641]: wsguardian2 not a cluster member after 0 sec post_fail_delay
  204. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] Members Left:
  205. Jan 25 15:02:54 wsguardian1 fenced[12641]: fencing node "wsguardian2"
  206. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ]     r(0) ip(192.168.253.2)  
  207. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] Members Joined:
  208. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] CLM CONFIGURATION CHANGE
  209. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] New Configuration:
  210. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ]     r(0) ip(192.168.253.1)  
  211. Jan 25 15:02:54 wsguardian1 kernel: dlm: closing connection to node 2
  212. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] Members Left:
  213. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] Members Joined:
  214. Jan 25 15:02:54 wsguardian1 openais[12622]: [SYNC ] This node is within the primary component and will provide service.
  215. Jan 25 15:02:54 wsguardian1 openais[12622]: [TOTEM] entering OPERATIONAL state.
  216. Jan 25 15:02:54 wsguardian1 openais[12622]: [CLM  ] got nodejoin message 192.168.253.1
  217. Jan 25 15:02:54 wsguardian1 openais[12622]: [CPG  ] got joinlist message from node 1
  218.  
  219.  
  220.  
  221. Log in node 2
  222.  
  223. Jan 25 15:02:43 wsguardian2 kernel: igb: eth4 NIC Link is Down
  224. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: PingAck did not arrive in time.
  225. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  226. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: asender terminated
  227. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: Terminating asender thread
  228. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: short read expecting header on sock: r=-512
  229. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: Creating new current UUID
  230. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: Connection closed
  231. Jan 25 15:02:52 wsguardian2 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  232. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  233. Jan 25 15:02:52 wsguardian2 rhcs_fence: 156; DEBUG: Found!
  234. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  235. Jan 25 15:02:52 wsguardian2 rhcs_fence: 128; DEBUG: It is not!
  236. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/sbin/fence_node]
  237. Jan 25 15:02:52 wsguardian2 rhcs_fence: 136; DEBUG: Found!
  238. Jan 25 15:02:52 wsguardian2 rhcs_fence: 146; DEBUG: Set 'path::fence_node' to: [/sbin/fence_node]
  239. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/usr/sbin/fence_node]
  240. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  241. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/bin/fence_node]
  242. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  243. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/usr/bin/fence_node]
  244. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  245. Jan 25 15:02:52 wsguardian2 rhcs_fence: 161; DEBUG: Checking again if: [fence_node] is at: [/sbin/fence_node].
  246. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  247. Jan 25 15:02:52 wsguardian2 rhcs_fence: 156; DEBUG: Found!
  248. Jan 25 15:02:52 wsguardian2 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  249. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_db]
  250. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [0]
  251. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian1]
  252. Jan 25 15:02:52 wsguardian2 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  253. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Version: 6.2.0
  254. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Config Version: 2
  255. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Name: WSGClust
  256. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Id: 21470
  257. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Member: Yes
  258. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Generation: 328
  259. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Membership state: Cluster-Member
  260. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Nodes: 2
  261. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Expected votes: 1
  262. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Total votes: 2
  263. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Node votes: 1
  264. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Quorum: 1  
  265. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Active subsystems: 7
  266. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Flags: 2node Dirty
  267. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Ports Bound: 0 11  
  268. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Node name: wsguardian2
  269. Jan 25 15:02:52 wsguardian2 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [0]
  270. Jan 25 15:02:52 wsguardian2 rhcs_fence: 471; DEBUG: I am: [wsguardian2]
  271. Jan 25 15:02:52 wsguardian2 rhcs_fence: 403; DEBUG: Checking the state of resource with minor number: [0]
  272. Jan 25 15:02:52 wsguardian2 rhcs_fence: 409; DEBUG: minor: [0]
  273. Jan 25 15:02:52 wsguardian2 rhcs_fence: 412; DEBUG: shell call: [</proc/drbd]
  274. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output: version: 8.3.8 (api:88/proto:86-94)
  275. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output: GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09
  276. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:  0: cs:NetworkFailure ro:Primary/Unknown ds:UpToDate/DUnknown C s----
  277. Jan 25 15:02:52 wsguardian2 rhcs_fence: 423; DEBUG: read state of minor: [0] as: [UpToDate]
  278. Jan 25 15:02:52 wsguardian2 rhcs_fence: 425; DEBUG: sys::local_res_uptodate: [1]
  279. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:     ns:500 nr:276 dw:776 dr:6340 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  280. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:  1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
  281. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:     ns:76 nr:68 dw:144 dr:6040 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  282. Jan 25 15:02:52 wsguardian2 rhcs_fence: 430; DEBUG: Attempt to collect UpToDate information device with minor: [0] exited with: [0]
  283. Jan 25 15:02:52 wsguardian2 rhcs_fence: 432; DEBUG: UpToDate: [1]
  284. Jan 25 15:02:52 wsguardian2 rhcs_fence: 367; DEBUG: shell call: [/usr/sbin/cman_tool -a -F id,name,type,addr nodes]
  285. Jan 25 15:02:52 wsguardian2 rhcs_fence: 374; DEBUG: id: [1], name: [wsguardian1], member: [M], address: [192.168.253.1]
  286. Jan 25 15:02:52 wsguardian2 rhcs_fence: 380; DEBUG: output: 1 wsguardian1 M 192.168.253.1
  287. Jan 25 15:02:52 wsguardian2 rhcs_fence: 374; DEBUG: id: [2], name: [wsguardian2], member: [M], address: [192.168.253.2]
  288. Jan 25 15:02:52 wsguardian2 rhcs_fence: 380; DEBUG: output: 2 wsguardian2 M 192.168.253.2
  289. Jan 25 15:02:52 wsguardian2 rhcs_fence: 384; DEBUG: Attempt to gather cluster member information exited with: [0]
  290. Jan 25 15:02:52 wsguardian2 rhcs_fence: 343; I have identified my target: [wsguardian1]
  291. Jan 25 15:02:52 wsguardian2 rhcs_fence: 191; DEBUG: I am: [wsguardian2] and my id is: [2]
  292. Jan 25 15:02:52 wsguardian2 rhcs_fence: 217; Delaying for: [9] seconds to avoid dual-fencing...
  293. Jan 25 15:02:52 wsguardian2 openais[12489]: [TOTEM] The token was lost in the OPERATIONAL state.
  294. Jan 25 15:02:52 wsguardian2 openais[12489]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
  295. Jan 25 15:02:52 wsguardian2 openais[12489]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
  296. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: PingAck did not arrive in time.
  297. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  298. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: asender terminated
  299. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: Terminating asender thread
  300. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: short read expecting header on sock: r=-512
  301. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: Creating new current UUID
  302. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: Connection closed
  303. Jan 25 15:02:52 wsguardian2 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  304. Jan 25 15:02:52 wsguardian2 openais[12489]: [TOTEM] entering GATHER state from 2.
  305. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  306. Jan 25 15:02:52 wsguardian2 rhcs_fence: 156; DEBUG: Found!
  307. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  308. Jan 25 15:02:52 wsguardian2 rhcs_fence: 128; DEBUG: It is not!
  309. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/sbin/fence_node]
  310. Jan 25 15:02:52 wsguardian2 rhcs_fence: 136; DEBUG: Found!
  311. Jan 25 15:02:52 wsguardian2 rhcs_fence: 146; DEBUG: Set 'path::fence_node' to: [/sbin/fence_node]
  312. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/usr/sbin/fence_node]
  313. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  314. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/bin/fence_node]
  315. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  316. Jan 25 15:02:52 wsguardian2 rhcs_fence: 133; DEBUG: Checking: [/usr/bin/fence_node]
  317. Jan 25 15:02:52 wsguardian2 rhcs_fence: 150; DEBUG: Not found!
  318. Jan 25 15:02:52 wsguardian2 rhcs_fence: 161; DEBUG: Checking again if: [fence_node] is at: [/sbin/fence_node].
  319. Jan 25 15:02:52 wsguardian2 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  320. Jan 25 15:02:52 wsguardian2 rhcs_fence: 156; DEBUG: Found!
  321. Jan 25 15:02:52 wsguardian2 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  322. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_config]
  323. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [1]
  324. Jan 25 15:02:52 wsguardian2 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian1]
  325. Jan 25 15:02:52 wsguardian2 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  326. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Version: 6.2.0
  327. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Config Version: 2
  328. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Name: WSGClust
  329. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Id: 21470
  330. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Member: Yes
  331. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Cluster Generation: 328
  332. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Membership state: Cluster-Member
  333. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Nodes: 2
  334. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Expected votes: 1
  335. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Total votes: 2
  336. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Node votes: 1
  337. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Quorum: 1  
  338. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Active subsystems: 7
  339. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Flags: 2node Dirty
  340. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Ports Bound: 0 11  
  341. Jan 25 15:02:52 wsguardian2 rhcs_fence: 460; DEBUG: output: Node name: wsguardian2
  342. Jan 25 15:02:52 wsguardian2 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [0]
  343. Jan 25 15:02:52 wsguardian2 rhcs_fence: 471; DEBUG: I am: [wsguardian2]
  344. Jan 25 15:02:52 wsguardian2 rhcs_fence: 403; DEBUG: Checking the state of resource with minor number: [1]
  345. Jan 25 15:02:52 wsguardian2 rhcs_fence: 409; DEBUG: minor: [1]
  346. Jan 25 15:02:52 wsguardian2 rhcs_fence: 412; DEBUG: shell call: [</proc/drbd]
  347. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output: version: 8.3.8 (api:88/proto:86-94)
  348. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output: GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09
  349. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:  0: cs:NetworkFailure ro:Primary/Unknown ds:UpToDate/DUnknown C s----
  350. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:     ns:500 nr:276 dw:776 dr:6340 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  351. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:  1: cs:NetworkFailure ro:Primary/Unknown ds:UpToDate/DUnknown C s----
  352. Jan 25 15:02:52 wsguardian2 rhcs_fence: 423; DEBUG: read state of minor: [1] as: [UpToDate]
  353. Jan 25 15:02:52 wsguardian2 rhcs_fence: 425; DEBUG: sys::local_res_uptodate: [1]
  354. Jan 25 15:02:52 wsguardian2 rhcs_fence: 418; DEBUG: output:     ns:76 nr:68 dw:144 dr:6040 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
  355. Jan 25 15:02:52 wsguardian2 rhcs_fence: 430; DEBUG: Attempt to collect UpToDate information device with minor: [1] exited with: [0]
  356. Jan 25 15:02:52 wsguardian2 rhcs_fence: 432; DEBUG: UpToDate: [1]
  357. Jan 25 15:02:52 wsguardian2 rhcs_fence: 367; DEBUG: shell call: [/usr/sbin/cman_tool -a -F id,name,type,addr nodes]
  358. Jan 25 15:02:52 wsguardian2 rhcs_fence: 374; DEBUG: id: [1], name: [wsguardian1], member: [M], address: [192.168.253.1]
  359. Jan 25 15:02:52 wsguardian2 rhcs_fence: 380; DEBUG: output: 1 wsguardian1 M 192.168.253.1
  360. Jan 25 15:02:52 wsguardian2 rhcs_fence: 374; DEBUG: id: [2], name: [wsguardian2], member: [M], address: [192.168.253.2]
  361. Jan 25 15:02:52 wsguardian2 rhcs_fence: 380; DEBUG: output: 2 wsguardian2 M 192.168.253.2
  362. Jan 25 15:02:52 wsguardian2 rhcs_fence: 384; DEBUG: Attempt to gather cluster member information exited with: [0]
  363. Jan 25 15:02:52 wsguardian2 rhcs_fence: 343; I have identified my target: [wsguardian1]
  364. Jan 25 15:02:52 wsguardian2 rhcs_fence: 191; DEBUG: I am: [wsguardian2] and my id is: [2]
  365. Jan 25 15:02:52 wsguardian2 rhcs_fence: 217; Delaying for: [9] seconds to avoid dual-fencing...
  366. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] entering GATHER state from 0.
  367. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] Creating commit token because I am the rep.
  368. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] Storing new sequence id for ring 14c
  369. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] entering COMMIT state.
  370. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] entering RECOVERY state.
  371. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] position [0] member 192.168.253.2:
  372. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] previous ring seq 328 rep 192.168.253.1
  373. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] aru b3 high delivered b3 received flag 1
  374. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] Did not need to originate any messages in recovery.
  375. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] Sending initial ORF token
  376. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] CLM CONFIGURATION CHANGE
  377. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] New Configuration:
  378. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ]     r(0) ip(192.168.253.2)  
  379. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] Members Left:
  380. Jan 25 15:02:54 wsguardian2 fenced[12508]: wsguardian1 not a cluster member after 0 sec post_fail_delay
  381. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ]     r(0) ip(192.168.253.1)  
  382. Jan 25 15:02:54 wsguardian2 fenced[12508]: fencing node "wsguardian1"
  383. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] Members Joined:
  384. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] CLM CONFIGURATION CHANGE
  385. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] New Configuration:
  386. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ]     r(0) ip(192.168.253.2)  
  387. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] Members Left:
  388. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] Members Joined:
  389. Jan 25 15:02:54 wsguardian2 openais[12489]: [SYNC ] This node is within the primary component and will provide service.
  390. Jan 25 15:02:54 wsguardian2 openais[12489]: [TOTEM] entering OPERATIONAL state.
  391. Jan 25 15:02:54 wsguardian2 openais[12489]: [CLM  ] got nodejoin message 192.168.253.2
  392. Jan 25 15:02:54 wsguardian2 kernel: dlm: closing connection to node 1
  393. Jan 25 15:02:54 wsguardian2 openais[12489]: [CPG  ] got joinlist message from node 2