API tools faq
paste
Advertisement
digimer

Untitled

digimer
Oct 23rd, 2013
321
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 56.95 KB | None | 0 0
raw download clone embed print report
  1. # This starts after node 1 was fenced by the 'drbdadm connect r0' from node 2. By this time, the node was back in the cluster but node 2's r1 was not connected yet. The mode on rhcs_fence was back to 755 at this point.
  2.  
  3. Oct 23 15:05:07 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  4. Oct 23 15:05:08 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  5. Oct 23 15:05:08 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  6. Oct 23 15:05:08 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  7. Oct 23 15:05:08 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  8. Oct 23 15:05:08 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  9. Oct 23 15:05:08 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  10. Oct 23 15:05:08 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  11. Oct 23 15:05:09 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  12. Oct 23 15:05:10 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  13. Oct 23 15:05:10 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  14. Oct 23 15:05:10 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  15. Oct 23 15:05:10 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  16. Oct 23 15:05:10 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  17. Oct 23 15:05:10 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  18. Oct 23 15:05:10 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  19. Oct 23 15:05:11 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  20. Oct 23 15:05:12 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  21. Oct 23 15:05:12 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  22. Oct 23 15:05:12 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  23. Oct 23 15:05:12 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  24. Oct 23 15:05:12 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  25. Oct 23 15:05:12 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  26. Oct 23 15:05:12 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  27. Oct 23 15:05:13 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  28. Oct 23 15:05:14 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  29. Oct 23 15:05:14 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  30. Oct 23 15:05:14 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  31. Oct 23 15:05:14 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  32. Oct 23 15:05:14 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  33. Oct 23 15:05:14 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  34. Oct 23 15:05:14 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  35. Oct 23 15:05:15 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  36. Oct 23 15:05:16 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  37. Oct 23 15:05:16 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  38. Oct 23 15:05:16 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  39. Oct 23 15:05:16 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  40. Oct 23 15:05:16 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  41. Oct 23 15:05:16 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  42. Oct 23 15:05:16 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  43. Oct 23 15:05:17 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  44. Oct 23 15:05:18 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  45. Oct 23 15:05:18 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  46. Oct 23 15:05:18 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  47. Oct 23 15:05:18 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  48. Oct 23 15:05:18 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  49. Oct 23 15:05:18 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  50. Oct 23 15:05:18 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  51. Oct 23 15:05:19 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  52. Oct 23 15:05:20 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  53. Oct 23 15:05:20 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  54. Oct 23 15:05:20 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  55. Oct 23 15:05:20 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  56. Oct 23 15:05:20 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  57. Oct 23 15:05:20 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  58. Oct 23 15:05:20 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  59. Oct 23 15:05:21 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  60. Oct 23 15:05:22 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  61. Oct 23 15:05:22 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  62. Oct 23 15:05:22 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  63. Oct 23 15:05:22 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  64. Oct 23 15:05:22 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  65. Oct 23 15:05:22 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  66. Oct 23 15:05:22 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  67. Oct 23 15:05:23 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  68. Oct 23 15:05:24 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  69. Oct 23 15:05:24 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  70. Oct 23 15:05:24 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  71. Oct 23 15:05:24 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  72. Oct 23 15:05:24 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  73. Oct 23 15:05:24 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  74. Oct 23 15:05:24 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  75. Oct 23 15:05:25 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  76. Oct 23 15:05:26 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  77. Oct 23 15:05:26 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  78. Oct 23 15:05:26 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  79. Oct 23 15:05:26 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  80. Oct 23 15:05:26 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  81. Oct 23 15:05:26 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  82. Oct 23 15:05:26 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  83. Oct 23 15:05:28 an-c05n01 clvmd: Cluster LVM daemon started - connected to CMAN
  84. Oct 23 15:07:29 an-c05n01 libvirtd: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
  85. Oct 23 15:07:29 an-c05n01 rgmanager[3520]: Service service:storage_an01 started
  86. Oct 23 15:07:29 an-c05n01 kernel: lo: Disabled Privacy Extensions
  87. Oct 23 15:08:05 an-c05n01 rgmanager[7023]: [script] script:gfs2: status of /etc/init.d/gfs2 failed (returned 3)
  88. Oct 23 15:08:05 an-c05n01 rgmanager[3520]: status on script "gfs2" returned 1 (generic error)
  89. Oct 23 15:08:05 an-c05n01 rgmanager[3520]: Stopping service service:storage_an01
  90. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: conn( WFConnection -> Disconnecting )
  91. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Discarding network configuration.
  92. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Connection closed
  93. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: conn( Disconnecting -> StandAlone )
  94. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: receiver terminated
  95. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Terminating drbd0_receiver
  96. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: disk( Consistent -> Failed )
  97. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  98. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  99. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: disk( Failed -> Diskless )
  100. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: drbd_bm_resize called with capacity == 0
  101. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: worker terminated
  102. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Terminating drbd0_worker
  103. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: conn( WFConnection -> Disconnecting )
  104. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Discarding network configuration.
  105. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Connection closed
  106. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: conn( Disconnecting -> StandAlone )
  107. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: receiver terminated
  108. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Terminating drbd1_receiver
  109. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: disk( Consistent -> Failed )
  110. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  111. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  112. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: disk( Failed -> Diskless )
  113. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: drbd_bm_resize called with capacity == 0
  114. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: worker terminated
  115. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Terminating drbd1_worker
  116. Oct 23 15:08:07 an-c05n01 kernel: drbd: module cleanup done.
  117. Oct 23 15:08:07 an-c05n01 rgmanager[3520]: Service service:storage_an01 is recovering
  118. Oct 23 15:08:07 an-c05n01 rgmanager[3520]: Recovering failed service service:storage_an01
  119. Oct 23 15:08:07 an-c05n01 kernel: drbd: initialized. Version: 8.3.15 (api:88/proto:86-97)
  120. Oct 23 15:08:07 an-c05n01 kernel: drbd: GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build64R6, 2012-12-20 20:09:51
  121. Oct 23 15:08:07 an-c05n01 kernel: drbd: registered as block device major 147
  122. Oct 23 15:08:07 an-c05n01 kernel: drbd: minor_table @ 0xffff880637cd3a80
  123. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Starting worker thread (from cqueue [3093])
  124. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: disk( Diskless -> Attaching )
  125. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Found 4 transactions (200 active extents) in activity log.
  126. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Method to ensure write ordering: flush
  127. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: max BIO size = 131072
  128. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: drbd_bm_resize called with capacity == 831061152
  129. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: resync bitmap: bits=103882644 words=1623167 pages=3171
  130. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: size = 396 GB (415530576 KB)
  131. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: bitmap READ of 3171 pages took 33 jiffies
  132. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: recounting of set bits took additional 15 jiffies
  133. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  134. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: disk( Attaching -> Consistent )
  135. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: attached to UUIDs C9F77959FD4A71D2:0000000000000000:282E11A241B8A05B:282D11A241B8A05B
  136. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Starting worker thread (from cqueue [3093])
  137. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: disk( Diskless -> Attaching )
  138. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: ASSERT( from_tnr - cnr + i - from == mx+1 ) in /home/phil/rpmbuild/BUILD/drbd-8.3.15/drbd/drbd_actlog.c:558
  139. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Found 4 transactions (248 active extents) in activity log.
  140. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Method to ensure write ordering: flush
  141. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: max BIO size = 131072
  142. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: drbd_bm_resize called with capacity == 830088384
  143. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: resync bitmap: bits=103761048 words=1621267 pages=3167
  144. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: size = 396 GB (415044192 KB)
  145. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: bitmap READ of 3167 pages took 44 jiffies
  146. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: recounting of set bits took additional 14 jiffies
  147. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  148. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: disk( Attaching -> Consistent )
  149. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: attached to UUIDs 8314C38D1738144A:0000000000000000:97927DD1274D9799:97917DD1274D9799
  150. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: conn( StandAlone -> Unconnected )
  151. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: Starting receiver thread (from drbd0_worker [7263])
  152. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: receiver (re)started
  153. Oct 23 15:08:07 an-c05n01 kernel: block drbd0: conn( Unconnected -> WFConnection )
  154. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: conn( StandAlone -> Unconnected )
  155. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: Starting receiver thread (from drbd1_worker [7281])
  156. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: receiver (re)started
  157. Oct 23 15:08:07 an-c05n01 kernel: block drbd1: conn( Unconnected -> WFConnection )
  158. Oct 23 15:13:07 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  159. Oct 23 15:13:08 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  160. Oct 23 15:13:08 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  161. Oct 23 15:13:08 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  162. Oct 23 15:13:08 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  163. Oct 23 15:13:08 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  164. Oct 23 15:13:08 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  165. Oct 23 15:13:08 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  166. Oct 23 15:13:09 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  167. Oct 23 15:13:10 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  168. Oct 23 15:13:10 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  169. Oct 23 15:13:10 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  170. Oct 23 15:13:10 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  171. Oct 23 15:13:10 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  172. Oct 23 15:13:10 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  173. Oct 23 15:13:10 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  174. Oct 23 15:13:11 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  175. Oct 23 15:13:12 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  176. Oct 23 15:13:12 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  177. Oct 23 15:13:12 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  178. Oct 23 15:13:12 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  179. Oct 23 15:13:12 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  180. Oct 23 15:13:12 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  181. Oct 23 15:13:12 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  182. Oct 23 15:13:13 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  183. Oct 23 15:13:14 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  184. Oct 23 15:13:14 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  185. Oct 23 15:13:14 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  186. Oct 23 15:13:14 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  187. Oct 23 15:13:14 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  188. Oct 23 15:13:14 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  189. Oct 23 15:13:14 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  190. Oct 23 15:13:15 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  191. Oct 23 15:13:16 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  192. Oct 23 15:13:16 an-c05n01 rhcs_fence: Local resource: [r0], minor: [0] is NOT 'UpToDate', will not fence peer.
  193. Oct 23 15:13:16 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 1 (0x100)
  194. Oct 23 15:13:16 an-c05n01 kernel: block drbd0: fence-peer helper broken, returned 1
  195. Oct 23 15:13:16 an-c05n01 kernel: block drbd0: State change failed: Need access to UpToDate data
  196. Oct 23 15:13:16 an-c05n01 kernel: block drbd0: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  197. Oct 23 15:13:16 an-c05n01 kernel: block drbd0: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  198. Oct 23 15:13:17 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  199. Oct 23 15:13:18 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  200. Oct 23 15:13:18 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  201. Oct 23 15:13:18 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  202. Oct 23 15:13:18 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  203. Oct 23 15:13:18 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  204. Oct 23 15:13:18 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  205. Oct 23 15:13:18 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  206. Oct 23 15:13:19 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  207. Oct 23 15:13:20 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  208. Oct 23 15:13:20 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  209. Oct 23 15:13:20 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  210. Oct 23 15:13:20 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  211. Oct 23 15:13:20 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  212. Oct 23 15:13:20 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  213. Oct 23 15:13:20 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  214. Oct 23 15:13:21 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  215. Oct 23 15:13:22 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  216. Oct 23 15:13:22 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  217. Oct 23 15:13:22 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  218. Oct 23 15:13:22 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  219. Oct 23 15:13:22 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  220. Oct 23 15:13:22 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  221. Oct 23 15:13:22 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  222. Oct 23 15:13:23 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  223. Oct 23 15:13:24 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  224. Oct 23 15:13:24 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  225. Oct 23 15:13:24 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  226. Oct 23 15:13:24 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  227. Oct 23 15:13:24 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  228. Oct 23 15:13:24 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  229. Oct 23 15:13:24 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  230. Oct 23 15:13:25 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  231. Oct 23 15:13:26 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  232. Oct 23 15:13:26 an-c05n01 rhcs_fence: Local resource: [r1], minor: [1] is NOT 'UpToDate', will not fence peer.
  233. Oct 23 15:13:26 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100)
  234. Oct 23 15:13:26 an-c05n01 kernel: block drbd1: fence-peer helper broken, returned 1
  235. Oct 23 15:13:26 an-c05n01 kernel: block drbd1: State change failed: Need access to UpToDate data
  236. Oct 23 15:13:26 an-c05n01 kernel: block drbd1: state = { cs:WFConnection ro:Secondary/Unknown ds:Consistent/DUnknown r----- }
  237. Oct 23 15:13:26 an-c05n01 kernel: block drbd1: wanted = { cs:WFConnection ro:Primary/Unknown ds:Consistent/DUnknown s---F- }
  238. Oct 23 15:13:28 an-c05n01 clvmd: Cluster LVM daemon started - connected to CMAN
  239. Oct 23 15:15:29 an-c05n01 libvirtd: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
  240. Oct 23 15:15:29 an-c05n01 rgmanager[3520]: Service service:storage_an01 started
  241. Oct 23 15:15:29 an-c05n01 kernel: lo: Disabled Privacy Extensions
  242. Oct 23 15:16:06 an-c05n01 rgmanager[8155]: [script] script:gfs2: status of /etc/init.d/gfs2 failed (returned 3)
  243. Oct 23 15:16:06 an-c05n01 rgmanager[3520]: status on script "gfs2" returned 1 (generic error)
  244. Oct 23 15:16:06 an-c05n01 rgmanager[3520]: Stopping service service:storage_an01
  245. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: conn( WFConnection -> Disconnecting )
  246. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Discarding network configuration.
  247. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Connection closed
  248. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: conn( Disconnecting -> StandAlone )
  249. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: receiver terminated
  250. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Terminating drbd0_receiver
  251. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: disk( Consistent -> Failed )
  252. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: bitmap WRITE of 0 pages took 1 jiffies
  253. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  254. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: disk( Failed -> Diskless )
  255. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: drbd_bm_resize called with capacity == 0
  256. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: worker terminated
  257. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Terminating drbd0_worker
  258. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: conn( WFConnection -> Disconnecting )
  259. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Discarding network configuration.
  260. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Connection closed
  261. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: conn( Disconnecting -> StandAlone )
  262. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: receiver terminated
  263. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Terminating drbd1_receiver
  264. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: disk( Consistent -> Failed )
  265. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  266. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  267. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: disk( Failed -> Diskless )
  268. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: drbd_bm_resize called with capacity == 0
  269. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: worker terminated
  270. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Terminating drbd1_worker
  271. Oct 23 15:16:07 an-c05n01 kernel: drbd: module cleanup done.
  272. Oct 23 15:16:07 an-c05n01 rgmanager[3520]: Service service:storage_an01 is recovering
  273. Oct 23 15:16:07 an-c05n01 rgmanager[3520]: Recovering failed service service:storage_an01
  274. Oct 23 15:16:07 an-c05n01 kernel: drbd: initialized. Version: 8.3.15 (api:88/proto:86-97)
  275. Oct 23 15:16:07 an-c05n01 kernel: drbd: GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build64R6, 2012-12-20 20:09:51
  276. Oct 23 15:16:07 an-c05n01 kernel: drbd: registered as block device major 147
  277. Oct 23 15:16:07 an-c05n01 kernel: drbd: minor_table @ 0xffff8806379b1180
  278. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Starting worker thread (from cqueue [3093])
  279. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: disk( Diskless -> Attaching )
  280. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Found 4 transactions (200 active extents) in activity log.
  281. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Method to ensure write ordering: flush
  282. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: max BIO size = 131072
  283. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: drbd_bm_resize called with capacity == 831061152
  284. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: resync bitmap: bits=103882644 words=1623167 pages=3171
  285. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: size = 396 GB (415530576 KB)
  286. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: bitmap READ of 3171 pages took 33 jiffies
  287. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: recounting of set bits took additional 15 jiffies
  288. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  289. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: disk( Attaching -> Consistent )
  290. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: attached to UUIDs C9F77959FD4A71D2:0000000000000000:282E11A241B8A05B:282D11A241B8A05B
  291. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Starting worker thread (from cqueue [3093])
  292. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: disk( Diskless -> Attaching )
  293. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: ASSERT( from_tnr - cnr + i - from == mx+1 ) in /home/phil/rpmbuild/BUILD/drbd-8.3.15/drbd/drbd_actlog.c:558
  294. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Found 4 transactions (248 active extents) in activity log.
  295. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Method to ensure write ordering: flush
  296. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: max BIO size = 131072
  297. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: drbd_bm_resize called with capacity == 830088384
  298. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: resync bitmap: bits=103761048 words=1621267 pages=3167
  299. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: size = 396 GB (415044192 KB)
  300. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: bitmap READ of 3167 pages took 44 jiffies
  301. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: recounting of set bits took additional 12 jiffies
  302. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  303. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: disk( Attaching -> Consistent )
  304. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: attached to UUIDs 8314C38D1738144A:0000000000000000:97927DD1274D9799:97917DD1274D9799
  305. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: conn( StandAlone -> Unconnected )
  306. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: Starting receiver thread (from drbd0_worker [8394])
  307. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: receiver (re)started
  308. Oct 23 15:16:07 an-c05n01 kernel: block drbd0: conn( Unconnected -> WFConnection )
  309. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: conn( StandAlone -> Unconnected )
  310. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: Starting receiver thread (from drbd1_worker [8413])
  311. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: receiver (re)started
  312. Oct 23 15:16:07 an-c05n01 kernel: block drbd1: conn( Unconnected -> WFConnection )
  313. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: Handshake successful: Agreed network protocol version 97
  314. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: conn( WFConnection -> WFReportParams )
  315. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: Starting asender thread (from drbd0_receiver [8459])
  316. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: data-integrity-alg: <not-used>
  317. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: drbd_sync_handshake:
  318. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: self C9F77959FD4A71D2:0000000000000000:282E11A241B8A05B:282D11A241B8A05B bits:0 flags:0
  319. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: peer BDB851199DCABE3F:C9F77959FD4A71D3:282E11A241B8A05A:282D11A241B8A05B bits:427 flags:0
  320. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: uuid_compare()=-1 by rule 50
  321. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
  322. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
  323. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: updated sync uuid C9F87959FD4A71D2:0000000000000000:282E11A241B8A05B:282D11A241B8A05B
  324. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
  325. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
  326. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
  327. Oct 23 15:16:19 an-c05n01 kernel: block drbd0: Began resync as SyncTarget (will sync 1708 KB [427 bits set]).
  328. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: Resync done (total 2 sec; paused 0 sec; 852 K/sec)
  329. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: updated UUIDs BDB851199DCABE3E:0000000000000000:C9F87959FD4A71D2:C9F77959FD4A71D3
  330. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
  331. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0
  332. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
  333. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: bitmap WRITE of 3153 pages took 9 jiffies
  334. Oct 23 15:16:22 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  335. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: Handshake successful: Agreed network protocol version 97
  336. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: conn( WFConnection -> WFReportParams )
  337. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: Starting asender thread (from drbd1_receiver [8464])
  338. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: data-integrity-alg: <not-used>
  339. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: drbd_sync_handshake:
  340. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: self 8314C38D1738144A:0000000000000000:97927DD1274D9799:97917DD1274D9799 bits:0 flags:0
  341. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: peer 8314C38D1738144B:0000000000000000:97927DD1274D9799:97917DD1274D9799 bits:0 flags:0
  342. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: uuid_compare()=1 by rule 40
  343. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapS ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> Consistent )
  344. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: meta connection shut down by peer.
  345. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: sock was shut down by peer
  346. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: sock_sendmsg returned -32
  347. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: peer( Primary -> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( Consistent -> DUnknown )
  348. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: short read expecting header on sock: r=0
  349. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: short sent ReportBitMap size=4096 sent=3708
  350. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  351. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  352. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: asender terminated
  353. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: Terminating drbd1_asender
  354. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: Connection closed
  355. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: conn( BrokenPipe -> Unconnected )
  356. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: receiver terminated
  357. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: Restarting drbd1_receiver
  358. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: receiver (re)started
  359. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: conn( Unconnected -> WFConnection )
  360. Oct 23 15:18:23 an-c05n01 kernel: block drbd0: role( Secondary -> Primary )
  361. Oct 23 15:18:23 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1
  362. Oct 23 15:18:24 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  363. Oct 23 15:18:34 an-c05n01 corosync[2730]: [TOTEM ] A processor failed, forming new configuration.
  364. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: PingAck did not arrive in time.
  365. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  366. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: asender terminated
  367. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: Terminating drbd0_asender
  368. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: Connection closed
  369. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: conn( NetworkFailure -> Unconnected )
  370. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: receiver terminated
  371. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: Restarting drbd0_receiver
  372. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: receiver (re)started
  373. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: conn( Unconnected -> WFConnection )
  374. Oct 23 15:18:34 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  375. Oct 23 15:18:35 an-c05n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...
  376. Oct 23 15:18:36 an-c05n01 corosync[2730]: [QUORUM] Members[1]: 1
  377. Oct 23 15:18:36 an-c05n01 corosync[2730]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  378. Oct 23 15:18:36 an-c05n01 dlm_controld[2813]: node_history_cluster_remove no nodeid 2
  379. Oct 23 15:18:36 an-c05n01 corosync[2730]: [CPG ] chosen downlist: sender r(0) ip(10.20.50.1) ; members(old:2 left:1)
  380. Oct 23 15:18:36 an-c05n01 corosync[2730]: [MAIN ] Completed service synchronization, ready to provide service.
  381. Oct 23 15:18:36 an-c05n01 kernel: dlm: closing connection to node 2
  382. Oct 23 15:18:36 an-c05n01 fenced[2787]: fencing node an-c05n02.alteeve.ca
  383. Oct 23 15:18:40 an-c05n01 fenced[2787]: fence an-c05n02.alteeve.ca success
  384. Oct 23 15:18:40 an-c05n01 fence_node[8831]: fence an-c05n02.alteeve.ca success
  385. Oct 23 15:18:40 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 7 (0x700)
  386. Oct 23 15:18:40 an-c05n01 kernel: block drbd1: fence-peer helper returned 7 (peer was stonithed)
  387. Oct 23 15:18:40 an-c05n01 kernel: block drbd1: role( Secondary -> Primary ) pdsk( DUnknown -> Outdated )
  388. Oct 23 15:18:40 an-c05n01 kernel: block drbd1: new current UUID FB2D0972D1C4DBAD:8314C38D1738144A:97927DD1274D9799:97917DD1274D9799
  389. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: Taking over service vm:vm01-dev from down member an-c05n02.alteeve.ca
  390. Oct 23 15:18:41 an-c05n01 rgmanager[9121]: [vm] Could not determine Hypervisor
  391. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: start on vm "vm01-dev" returned 2 (invalid argument(s))
  392. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: #68: Failed to start vm:vm01-dev; return value: 1
  393. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: Stopping service vm:vm01-dev
  394. Oct 23 15:18:41 an-c05n01 rgmanager[9159]: [vm] Could not determine Hypervisor
  395. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: stop on vm "vm01-dev" returned 2 (invalid argument(s))
  396. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: #12: RG vm:vm01-dev failed to stop; intervention required
  397. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: Service vm:vm01-dev is failed
  398. Oct 23 15:18:41 an-c05n01 rgmanager[3520]: #13: Service vm:vm01-dev failed to stop cleanly
  399. Oct 23 15:18:41 an-c05n01 clvmd: Cluster LVM daemon started - connected to CMAN
  400. Oct 23 15:18:42 an-c05n01 fence_node[8988]: fence an-c05n02.alteeve.ca success
  401. Oct 23 15:18:42 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 7 (0x700)
  402. Oct 23 15:18:42 an-c05n01 kernel: block drbd0: fence-peer helper returned 7 (peer was stonithed)
  403. Oct 23 15:18:42 an-c05n01 kernel: block drbd0: pdsk( DUnknown -> Outdated )
  404. Oct 23 15:18:42 an-c05n01 kernel: block drbd0: new current UUID 0EE0BDF5981A9549:BDB851199DCABE3F:C9F87959FD4A71D2:C9F77959FD4A71D3
  405. Oct 23 15:18:42 an-c05n01 kernel: block drbd0: susp( 1 -> 0 )
  406. Oct 23 15:18:42 an-c05n01 kernel: Slow work thread pool: Starting up
  407. Oct 23 15:18:42 an-c05n01 kernel: Slow work thread pool: Ready
  408. Oct 23 15:18:42 an-c05n01 kernel: GFS2 (built Oct 16 2013 18:39:46) installed
  409. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "an-cluster-A:shared"
  410. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: Joined cluster. Now mounting FS...
  411. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=0, already locked for use
  412. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=0: Looking at journal...
  413. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=0: Done
  414. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=1: Trying to acquire journal lock...
  415. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=1: Looking at journal...
  416. Oct 23 15:18:42 an-c05n01 kernel: GFS2: fsid=an-cluster-A:shared.0: jid=1: Done
  417. Oct 23 15:18:42 an-c05n01 libvirtd: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
  418. Oct 23 15:18:42 an-c05n01 rgmanager[3520]: Service service:storage_an01 started
  419. Oct 23 15:18:43 an-c05n01 kernel: lo: Disabled Privacy Extensions
  420. Oct 23 15:20:25 an-c05n01 corosync[2730]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  421. Oct 23 15:20:25 an-c05n01 corosync[2730]: [QUORUM] Members[2]: 1 2
  422. Oct 23 15:20:25 an-c05n01 corosync[2730]: [QUORUM] Members[2]: 1 2
  423. Oct 23 15:20:25 an-c05n01 corosync[2730]: [CPG ] chosen downlist: sender r(0) ip(10.20.50.1) ; members(old:1 left:0)
  424. Oct 23 15:20:25 an-c05n01 corosync[2730]: [MAIN ] Completed service synchronization, ready to provide service.
  425. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: Handshake successful: Agreed network protocol version 97
  426. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: conn( WFConnection -> WFReportParams )
  427. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: Starting asender thread (from drbd0_receiver [8459])
  428. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: data-integrity-alg: <not-used>
  429. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: drbd_sync_handshake:
  430. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: self 0EE0BDF5981A9549:BDB851199DCABE3F:C9F87959FD4A71D2:C9F77959FD4A71D3 bits:0 flags:0
  431. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: peer BDB851199DCABE3E:0000000000000000:C9F87959FD4A71D3:C9F77959FD4A71D3 bits:28672 flags:2
  432. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: uuid_compare()=1 by rule 70
  433. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )
  434. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: Handshake successful: Agreed network protocol version 97
  435. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: conn( WFConnection -> WFReportParams )
  436. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: Starting asender thread (from drbd1_receiver [8464])
  437. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: data-integrity-alg: <not-used>
  438. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: drbd_sync_handshake:
  439. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: self FB2D0972D1C4DBAD:8314C38D1738144A:97927DD1274D9799:97917DD1274D9799 bits:0 flags:0
  440. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: peer 8314C38D1738144A:0000000000000000:97927DD1274D9799:97917DD1274D9799 bits:44032 flags:2
  441. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: uuid_compare()=1 by rule 70
  442. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )
  443. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: peer( Secondary -> Primary )
  444. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: peer( Secondary -> Primary )
  445. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
  446. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
  447. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
  448. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: Began resync as SyncSource (will sync 114688 KB [28672 bits set]).
  449. Oct 23 15:20:35 an-c05n01 kernel: block drbd0: updated sync UUID 0EE0BDF5981A9549:BDB951199DCABE3F:BDB851199DCABE3F:C9F87959FD4A71D2
  450. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1
  451. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
  452. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
  453. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: Began resync as SyncSource (will sync 176128 KB [44032 bits set]).
  454. Oct 23 15:20:35 an-c05n01 kernel: block drbd1: updated sync UUID FB2D0972D1C4DBAD:8315C38D1738144A:8314C38D1738144A:97927DD1274D9799
  455. Oct 23 15:20:36 an-c05n01 kernel: dlm: got connection from 2
  456. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: peer( Primary -> Secondary )
  457. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: peer( Secondary -> Unknown ) conn( SyncSource -> TearDown )
  458. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: asender terminated
  459. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: Terminating drbd0_asender
  460. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: bitmap WRITE of 3171 pages took 11 jiffies
  461. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: 105 MB (26848 bits) marked out-of-sync by on disk bit-map.
  462. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: Connection closed
  463. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: conn( TearDown -> Unconnected )
  464. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: receiver terminated
  465. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: Restarting drbd0_receiver
  466. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: receiver (re)started
  467. Oct 23 15:20:41 an-c05n01 kernel: block drbd0: conn( Unconnected -> WFConnection )
  468. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: peer( Primary -> Secondary )
  469. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: peer( Secondary -> Unknown ) conn( SyncSource -> TearDown )
  470. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: asender terminated
  471. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: Terminating drbd1_asender
  472. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: bitmap WRITE of 3166 pages took 9 jiffies
  473. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: 165 MB (42240 bits) marked out-of-sync by on disk bit-map.
  474. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: Connection closed
  475. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: conn( TearDown -> Unconnected )
  476. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: receiver terminated
  477. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: Restarting drbd1_receiver
  478. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: receiver (re)started
  479. Oct 23 15:20:41 an-c05n01 kernel: block drbd1: conn( Unconnected -> WFConnection )
  480. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: Handshake successful: Agreed network protocol version 97
  481. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: conn( WFConnection -> WFReportParams )
  482. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: Starting asender thread (from drbd0_receiver [8459])
  483. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: data-integrity-alg: <not-used>
  484. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: drbd_sync_handshake:
  485. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: self 0EE0BDF5981A9549:BDB951199DCABE3F:BDB851199DCABE3F:C9F87959FD4A71D2 bits:26848 flags:0
  486. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: peer BDB951199DCABE3E:0000000000000000:C9F87959FD4A71D3:C9F77959FD4A71D3 bits:26848 flags:2
  487. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: uuid_compare()=1 by rule 70
  488. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: Becoming sync source due to disk states.
  489. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )
  490. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: Handshake successful: Agreed network protocol version 97
  491. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: conn( WFConnection -> WFReportParams )
  492. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: Starting asender thread (from drbd1_receiver [8464])
  493. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: data-integrity-alg: <not-used>
  494. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: drbd_sync_handshake:
  495. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: self FB2D0972D1C4DBAD:8315C38D1738144A:8314C38D1738144A:97927DD1274D9799 bits:42240 flags:0
  496. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: peer 8315C38D1738144A:0000000000000000:97927DD1274D9799:97917DD1274D9799 bits:42240 flags:2
  497. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: uuid_compare()=1 by rule 70
  498. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: Becoming sync source due to disk states.
  499. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )
  500. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: peer( Secondary -> Primary )
  501. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: peer( Secondary -> Primary )
  502. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
  503. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
  504. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: conn( WFBitMapS -> SyncSource )
  505. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: Began resync as SyncSource (will sync 107392 KB [26848 bits set]).
  506. Oct 23 15:20:53 an-c05n01 kernel: block drbd0: updated sync UUID 0EE0BDF5981A9549:BDBA51199DCABE3F:BDB951199DCABE3F:BDB851199DCABE3F
  507. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1
  508. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
  509. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: conn( WFBitMapS -> SyncSource )
  510. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: Began resync as SyncSource (will sync 168960 KB [42240 bits set]).
  511. Oct 23 15:20:53 an-c05n01 kernel: block drbd1: updated sync UUID FB2D0972D1C4DBAD:8316C38D1738144A:8315C38D1738144A:8314C38D1738144A
  512. Oct 23 15:20:59 an-c05n01 kernel: block drbd0: Resync done (total 5 sec; paused 0 sec; 21476 K/sec)
  513. Oct 23 15:20:59 an-c05n01 kernel: block drbd0: updated UUIDs 0EE0BDF5981A9549:0000000000000000:BDBA51199DCABE3F:BDB951199DCABE3F
  514. Oct 23 15:20:59 an-c05n01 kernel: block drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
  515. Oct 23 15:20:59 an-c05n01 kernel: block drbd0: bitmap WRITE of 3151 pages took 12 jiffies
  516. Oct 23 15:20:59 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  517. Oct 23 15:21:03 an-c05n01 kernel: block drbd1: Resync done (total 9 sec; paused 0 sec; 18772 K/sec)
  518. Oct 23 15:21:03 an-c05n01 kernel: block drbd1: updated UUIDs FB2D0972D1C4DBAD:0000000000000000:8316C38D1738144A:8315C38D1738144A
  519. Oct 23 15:21:03 an-c05n01 kernel: block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
  520. Oct 23 15:21:03 an-c05n01 kernel: block drbd1: bitmap WRITE of 3139 pages took 9 jiffies
  521. Oct 23 15:21:03 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  522. Oct 23 15:37:10 an-c05n01 rgmanager[3520]: Shutting down
  523. Oct 23 15:37:10 an-c05n01 rgmanager[3520]: Stopping service service:storage_an01
  524. Oct 23 15:37:11 an-c05n01 kernel: Slow work thread pool: Shutting down
  525. Oct 23 15:37:11 an-c05n01 kernel: Slow work thread pool: Shut down complete
  526. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: peer( Primary -> Secondary )
  527. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: State change failed: Refusing to be Primary while peer is not outdated
  528. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r----- }
  529. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: wanted = { cs:TearDown ro:Primary/Unknown ds:UpToDate/DUnknown s---F- }
  530. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> Outdated )
  531. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: new current UUID DE746785B6ABF68F:0EE0BDF5981A9549:BDBA51199DCABE3F:BDB951199DCABE3F
  532. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: asender terminated
  533. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: Terminating drbd0_asender
  534. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: Connection closed
  535. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: conn( TearDown -> Unconnected )
  536. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: receiver terminated
  537. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: Restarting drbd0_receiver
  538. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: receiver (re)started
  539. Oct 23 15:37:12 an-c05n01 kernel: block drbd0: conn( Unconnected -> WFConnection )
  540. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: peer( Primary -> Secondary )
  541. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: State change failed: Refusing to be Primary while peer is not outdated
  542. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r----- }
  543. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: wanted = { cs:TearDown ro:Primary/Unknown ds:UpToDate/DUnknown s---F- }
  544. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> Outdated )
  545. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: new current UUID E4DF757655A5C081:FB2D0972D1C4DBAD:8316C38D1738144A:8315C38D1738144A
  546. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: asender terminated
  547. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: Terminating drbd1_asender
  548. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: Connection closed
  549. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: conn( TearDown -> Unconnected )
  550. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: receiver terminated
  551. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: Restarting drbd1_receiver
  552. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: receiver (re)started
  553. Oct 23 15:37:12 an-c05n01 kernel: block drbd1: conn( Unconnected -> WFConnection )
  554. Oct 23 15:37:12 an-c05n01 rgmanager[3520]: Member 2 shutting down
  555. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: role( Primary -> Secondary )
  556. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  557. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  558. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: conn( WFConnection -> Disconnecting )
  559. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: Discarding network configuration.
  560. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: Connection closed
  561. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: conn( Disconnecting -> StandAlone )
  562. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: receiver terminated
  563. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: Terminating drbd0_receiver
  564. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: disk( UpToDate -> Failed )
  565. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  566. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  567. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: disk( Failed -> Diskless )
  568. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: drbd_bm_resize called with capacity == 0
  569. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: worker terminated
  570. Oct 23 15:39:12 an-c05n01 kernel: block drbd0: Terminating drbd0_worker
  571. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: role( Primary -> Secondary )
  572. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  573. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  574. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: conn( WFConnection -> Disconnecting )
  575. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: Discarding network configuration.
  576. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: Connection closed
  577. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: conn( Disconnecting -> StandAlone )
  578. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: receiver terminated
  579. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: Terminating drbd1_receiver
  580. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: disk( UpToDate -> Failed )
  581. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  582. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  583. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: disk( Failed -> Diskless )
  584. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: drbd_bm_resize called with capacity == 0
  585. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: worker terminated
  586. Oct 23 15:39:12 an-c05n01 kernel: block drbd1: Terminating drbd1_worker
  587. Oct 23 15:39:12 an-c05n01 kernel: drbd: module cleanup done.
  588. Oct 23 15:39:12 an-c05n01 rgmanager[3520]: Service service:storage_an01 is stopped
  589. Oct 23 15:39:12 an-c05n01 rgmanager[3520]: Disconnecting from CMAN
  590. Oct 23 15:39:27 an-c05n01 rgmanager[3520]: Exiting
  591. Oct 23 15:40:07 an-c05n01 fenced[2787]: cpg_mcast_joined error 12 handle 22221a7000000001 start
  592. Oct 23 15:40:09 an-c05n01 kernel: dlm: closing connection to node 2
  593. Oct 23 15:40:09 an-c05n01 kernel: dlm: closing connection to node 1
  594. Oct 23 15:40:09 an-c05n01 corosync[2730]: [QUORUM] Members[1]: 1
  595. Oct 23 15:40:09 an-c05n01 corosync[2730]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  596. Oct 23 15:40:09 an-c05n01 corosync[2730]: [CPG ] chosen downlist: sender r(0) ip(10.20.50.1) ; members(old:2 left:1)
  597. Oct 23 15:40:09 an-c05n01 corosync[2730]: [MAIN ] Completed service synchronization, ready to provide service.
  598. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Unloading all Corosync service engines.
  599. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync extended virtual synchrony service
  600. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync configuration service
  601. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync cluster closed process group service v1.01
  602. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync cluster config database access v1.01
  603. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync profile loading service
  604. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: openais checkpoint service B.01.01
  605. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync CMAN membership service 2.90
  606. Oct 23 15:40:09 an-c05n01 corosync[2730]: [SERV ] Service engine unloaded: corosync cluster quorum service v0.1
  607. Oct 23 15:40:09 an-c05n01 corosync[2730]: [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:1894.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!