API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Nov 27th, 2013
79
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 329.69 KB | None | 0 0
raw download clone embed print report
  1. Nov 27 01:13:11 vs001 cib: [1012]: info: cib_stats: Processed 254 operations (393.00us average, 0% utilization) in the last 10min
  2. Nov 27 01:17:01 vs001 CRON[29581]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
  3. Nov 27 01:18:36 vs001 kernel: [44148.617604] block drbd4: PingAck did not arrive in time.
  4. Nov 27 01:18:36 vs001 kernel: [44148.617645] block drbd4: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  5. Nov 27 01:18:36 vs001 kernel: [44148.621573] block drbd16: PingAck did not arrive in time.
  6. Nov 27 01:18:36 vs001 kernel: [44148.621624] block drbd16: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  7. Nov 27 01:18:36 vs001 kernel: [44148.717852] block drbd4: asender terminated
  8. Nov 27 01:18:36 vs001 kernel: [44148.717858] block drbd4: Terminating drbd4_asender
  9. Nov 27 01:18:36 vs001 kernel: [44148.717890] block drbd16: asender terminated
  10. Nov 27 01:18:36 vs001 kernel: [44148.717897] block drbd16: Terminating drbd16_asender
  11. Nov 27 01:18:36 vs001 kernel: [44148.717924] block drbd4: Connection closed
  12. Nov 27 01:18:36 vs001 kernel: [44148.717932] block drbd4: conn( NetworkFailure -> Unconnected )
  13. Nov 27 01:18:36 vs001 kernel: [44148.717939] block drbd4: receiver terminated
  14. Nov 27 01:18:36 vs001 kernel: [44148.717942] block drbd4: Restarting drbd4_receiver
  15. Nov 27 01:18:36 vs001 kernel: [44148.717945] block drbd4: receiver (re)started
  16. Nov 27 01:18:36 vs001 kernel: [44148.717952] block drbd4: conn( Unconnected -> WFConnection )
  17. Nov 27 01:18:36 vs001 kernel: [44148.717954] block drbd16: Connection closed
  18. Nov 27 01:18:36 vs001 kernel: [44148.717960] block drbd16: conn( NetworkFailure -> Unconnected )
  19. Nov 27 01:18:36 vs001 kernel: [44148.717966] block drbd16: receiver terminated
  20. Nov 27 01:18:36 vs001 kernel: [44148.717969] block drbd16: Restarting drbd16_receiver
  21. Nov 27 01:18:36 vs001 kernel: [44148.717972] block drbd16: receiver (re)started
  22. Nov 27 01:18:36 vs001 kernel: [44148.717977] block drbd16: conn( Unconnected -> WFConnection )
  23. Nov 27 01:18:37 vs001 kernel: [44149.441584] block drbd21: PingAck did not arrive in time.
  24. Nov 27 01:18:37 vs001 kernel: [44149.441625] block drbd21: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  25. Nov 27 01:18:37 vs001 kernel: [44149.516602] block drbd21: asender terminated
  26. Nov 27 01:18:37 vs001 kernel: [44149.516608] block drbd21: Terminating drbd21_asender
  27. Nov 27 01:18:37 vs001 kernel: [44149.516792] block drbd21: Connection closed
  28. Nov 27 01:18:37 vs001 kernel: [44149.516800] block drbd21: conn( NetworkFailure -> Unconnected )
  29. Nov 27 01:18:37 vs001 kernel: [44149.516805] block drbd21: receiver terminated
  30. Nov 27 01:18:37 vs001 kernel: [44149.516807] block drbd21: Restarting drbd21_receiver
  31. Nov 27 01:18:37 vs001 kernel: [44149.516810] block drbd21: receiver (re)started
  32. Nov 27 01:18:37 vs001 kernel: [44149.516814] block drbd21: conn( Unconnected -> WFConnection )
  33. Nov 27 01:18:37 vs001 corosync[3280]: [TOTEM ] A processor failed, forming new configuration.
  34. Nov 27 01:18:41 vs001 cib: [1012]: notice: ais_dispatch_message: Membership 740: quorum lost
  35. Nov 27 01:18:41 vs001 crmd: [1016]: notice: ais_dispatch_message: Membership 740: quorum lost
  36. Nov 27 01:18:41 vs001 cib: [1012]: info: crm_update_peer: Node vs002: id=687795291 state=lost (new) addr=r(0) ip(91.236.254.40) votes=1 born=560 seen=736 proc=00000000000000000000000000111312
  37. Nov 27 01:18:41 vs001 crmd: [1016]: info: ais_status_callback: status: vs002 is now lost (was member)
  38. Nov 27 01:18:41 vs001 crmd: [1016]: info: crm_update_peer: Node vs002: id=687795291 state=lost (new) addr=r(0) ip(91.236.254.40) votes=1 born=560 seen=736 proc=00000000000000000000000000111312
  39. Nov 27 01:18:41 vs001 cib: [1012]: info: crm_update_peer: Node vs003: id=721349723 state=lost (new) addr=r(0) ip(91.236.254.42) votes=1 born=148 seen=736 proc=00000000000000000000000000111312
  40. Nov 27 01:18:41 vs001 crmd: [1016]: info: ais_status_callback: status: vs003 is now lost (was member)
  41. Nov 27 01:18:41 vs001 crmd: [1016]: info: crm_update_peer: Node vs003: id=721349723 state=lost (new) addr=r(0) ip(91.236.254.42) votes=1 born=148 seen=736 proc=00000000000000000000000000111312
  42. Nov 27 01:18:41 vs001 crmd: [1016]: WARN: check_dead_member: Our DC node (vs003) left the cluster
  43. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 740: memb=1, new=0, lost=2
  44. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: memb: vs001 603909211
  45. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: lost: vs002 687795291
  46. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: lost: vs003 721349723
  47. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 740: memb=1, new=0, lost=0
  48. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: MEMB: vs001 603909211
  49. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: ais_mark_unseen_peer_dead: Node vs002 was not seen in the previous transition
  50. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: update_member: Node 687795291/vs002 is now: lost
  51. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: ais_mark_unseen_peer_dead: Node vs003 was not seen in the previous transition
  52. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: update_member: Node 721349723/vs003 is now: lost
  53. Nov 27 01:18:41 vs001 corosync[3280]: [pcmk ] info: send_member_notification: Sending membership update 740 to 2 children
  54. Nov 27 01:18:41 vs001 corosync[3280]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  55. Nov 27 01:18:41 vs001 corosync[3280]: [CPG ] chosen downlist: sender r(0) ip(91.236.254.35) ; members(old:3 left:2)
  56. Nov 27 01:18:41 vs001 corosync[3280]: [MAIN ] Completed service synchronization, ready to provide service.
  57. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]
  58. Nov 27 01:18:41 vs001 crmd: [1016]: info: update_dc: Unset DC vs003
  59. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
  60. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_te_control: Registering TE UUID: 2a9d944e-4153-417a-a3c1-e9a5c461707d
  61. Nov 27 01:18:41 vs001 crmd: [1016]: info: set_graph_functions: Setting custom graph functions
  62. Nov 27 01:18:41 vs001 crmd: [1016]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
  63. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_dc_takeover: Taking over DC status for this partition
  64. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_readwrite: We are now in R/W mode
  65. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/122, version=0.629.289): ok (rc=0)
  66. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/123, version=0.629.290): ok (rc=0)
  67. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/125, version=0.629.291): ok (rc=0)
  68. Nov 27 01:18:41 vs001 crmd: [1016]: info: join_make_offer: Making join offers based on membership 740
  69. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
  70. Nov 27 01:18:41 vs001 crmd: [1016]: info: ais_dispatch_message: Membership 740: quorum still lost
  71. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/127, version=0.629.292): ok (rc=0)
  72. Nov 27 01:18:41 vs001 crmd: [1016]: info: crmd_ais_dispatch: Setting expected votes to 3
  73. Nov 27 01:18:41 vs001 crmd: [1016]: info: update_dc: Set DC to vs001 (3.0.5)
  74. Nov 27 01:18:41 vs001 crmd: [1016]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
  75. Nov 27 01:18:41 vs001 crmd: [1016]: info: config_query_callback: Checking for expired actions every 900000ms
  76. Nov 27 01:18:41 vs001 crmd: [1016]: info: config_query_callback: Sending expected-votes=3 to corosync
  77. Nov 27 01:18:41 vs001 crmd: [1016]: info: ais_dispatch_message: Membership 740: quorum still lost
  78. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/130, version=0.629.293): ok (rc=0)
  79. Nov 27 01:18:41 vs001 crmd: [1016]: info: crmd_ais_dispatch: Setting expected votes to 3
  80. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/133, version=0.629.294): ok (rc=0)
  81. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
  82. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
  83. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_dc_join_finalize: join-1: Syncing the CIB from vs001 to the rest of the cluster
  84. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/134, version=0.629.294): ok (rc=0)
  85. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/135, version=0.629.295): ok (rc=0)
  86. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_dc_join_ack: join-1: Updating node state to member for vs001
  87. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vs001']/lrm (origin=local/crmd/136, version=0.629.296): ok (rc=0)
  88. Nov 27 01:18:41 vs001 crmd: [1016]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vs001']/lrm": ok (rc=0)
  89. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
  90. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
  91. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
  92. Nov 27 01:18:41 vs001 crmd: [1016]: info: crm_update_quorum: Updating quorum status to false (call=140)
  93. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
  94. Nov 27 01:18:41 vs001 crmd: [1016]: info: abort_transition_graph: do_te_invoke:167 - Triggered transition abort (complete=1) : Peer Cancelled
  95. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_ronon:1 (10000)
  96. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_pe_invoke: Query 141: Requesting the current CIB: S_POLICY_ENGINE
  97. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/138, version=0.629.298): ok (rc=0)
  98. Nov 27 01:18:41 vs001 crmd: [1016]: WARN: match_down_event: No match for shutdown action on vs003
  99. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_update_diff: Stonith/shutdown of vs003 not matched
  100. Nov 27 01:18:41 vs001 crmd: [1016]: info: abort_transition_graph: te_update_diff:215 - Triggered transition abort (complete=1, tag=node_state, id=vs003, magic=NA, cib=0.629.299) : Node failure
  101. Nov 27 01:18:41 vs001 crmd: [1016]: WARN: match_down_event: No match for shutdown action on vs002
  102. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_update_diff: Stonith/shutdown of vs002 not matched
  103. Nov 27 01:18:41 vs001 crmd: [1016]: info: abort_transition_graph: te_update_diff:215 - Triggered transition abort (complete=1, tag=node_state, id=vs002, magic=NA, cib=0.629.299) : Node failure
  104. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_pe_invoke: Query 142: Requesting the current CIB: S_POLICY_ENGINE
  105. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_pe_invoke: Query 143: Requesting the current CIB: S_POLICY_ENGINE
  106. Nov 27 01:18:41 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/140, version=0.629.300): ok (rc=0)
  107. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup:0 (10000)
  108. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-vg_cinder-HDD1 (INFINITY)
  109. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc-data:1 (10000)
  110. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard-data:0 (10000)
  111. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_oneill:1 (10000)
  112. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_pe_invoke_callback: Invoking the PE: query=143, ref=pe_calc-dc-1385511521-83, seq=740, quorate=0
  113. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter:1 (10000)
  114. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
  115. Nov 27 01:18:41 vs001 pengine: [1015]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled
  116. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup-data:1 (10000)
  117. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_backup:0_last_failure_0 found resource drbd_backup:0 active on vs001
  118. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_sheppard-data:0_last_failure_0 found resource drbd_sheppard-data:0 active on vs001
  119. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_carter:0_last_failure_0 found resource drbd_carter:0 active on vs001
  120. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_oneill:0_last_failure_0 found resource drbd_oneill:0 active on vs001
  121. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_tealc:0_last_failure_0 found resource drbd_tealc:0 active on vs001
  122. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_carter-data:0_last_failure_0 found resource drbd_carter-data:0 active on vs001
  123. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_rush:0_last_failure_0 found resource drbd_rush:0 active on vs001
  124. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_backup-data:0_last_failure_0 found resource drbd_backup-data:0 active on vs001
  125. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_tealc-data:0_last_failure_0 found resource drbd_tealc-data:0 active on vs001
  126. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_sheppard:0_last_failure_0 found resource drbd_sheppard:0 active on vs001
  127. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_ronon:0_last_failure_0 found resource drbd_ronon:0 active on vs001
  128. Nov 27 01:18:41 vs001 pengine: [1015]: WARN: unpack_rsc_op: Processing failed op vg_cinder-HDD1_last_failure_0 on vs001: not running (7)
  129. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_hammond:0_last_failure_0 found resource drbd_hammond:2 active on vs001
  130. Nov 27 01:18:41 vs001 pengine: [1015]: notice: unpack_rsc_op: Operation drbd_cinder-HDD1:0_last_failure_0 found resource drbd_cinder-HDD1:2 active in master mode on vs001
  131. Nov 27 01:18:41 vs001 pengine: [1015]: WARN: common_apply_stickiness: Forcing vg_cinder-HDD1 away from vs001 after 1000000 failures (max=1000000)
  132. Nov 27 01:18:41 vs001 pengine: [1015]: WARN: common_apply_stickiness: Forcing vg_cinder-HDD1 away from vs003 after 1000000 failures (max=1000000)
  133. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_rush:1 on vs001
  134. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_rush:1 on vs001
  135. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_ronon:1 on vs001
  136. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_ronon:1 on vs001
  137. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_hammond:1 on vs001
  138. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_hammond:1 on vs001
  139. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_carter:1 on vs001
  140. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_carter:1 on vs001
  141. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_carter-data:1 on vs001
  142. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_carter-data:1 on vs001
  143. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_oneill:1 on vs001
  144. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_oneill:1 on vs001
  145. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_backup:0 on vs001
  146. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_backup:0 on vs001
  147. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_backup-data:1 on vs001
  148. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_backup-data:1 on vs001
  149. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_sheppard:0 on vs001
  150. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_sheppard:0 on vs001
  151. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_sheppard-data:0 on vs001
  152. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_sheppard-data:0 on vs001
  153. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_tealc:1 on vs001
  154. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_tealc:1 on vs001
  155. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_tealc-data:1 on vs001
  156. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_tealc-data:1 on vs001
  157. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_cinder-HDD1:1 on vs001
  158. Nov 27 01:18:41 vs001 pengine: [1015]: notice: RecurringOp: Start recurring monitor (29s) for drbd_cinder-HDD1:1 on vs001
  159. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter-data:1 (10000)
  160. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_rush:0#011(Stopped)
  161. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_rush:1#011(Slave -> Master vs001)
  162. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_rush#011(vs001)
  163. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_rush#011(vs001)
  164. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_ronon:0#011(Stopped)
  165. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_ronon:1#011(Slave -> Master vs001)
  166. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_ronon#011(vs001)
  167. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_ronon#011(vs001)
  168. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_hammond:0#011(Stopped)
  169. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_hammond:1#011(Slave -> Master vs001)
  170. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_hammond#011(vs001)
  171. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_hammond#011(vs001)
  172. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_carter:0#011(Stopped)
  173. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_carter:1#011(Slave -> Master vs001)
  174. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_carter-data:0#011(Stopped)
  175. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_carter-data:1#011(Slave -> Master vs001)
  176. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_carter#011(vs001)
  177. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_carter#011(vs001)
  178. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_oneill:0#011(Stopped)
  179. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_oneill:1#011(Slave -> Master vs001)
  180. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_oneill#011(vs001)
  181. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_oneill#011(vs001)
  182. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_backup:0#011(Slave -> Master vs001)
  183. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_backup:1#011(Stopped)
  184. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_backup-data:0#011(Stopped)
  185. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_backup-data:1#011(Slave -> Master vs001)
  186. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_backup#011(vs001)
  187. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_backup-data#011(vs001)
  188. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_backup#011(vs001)
  189. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_mckay:0#011(Stopped)
  190. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_mckay:1#011(Stopped)
  191. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave fs_mckay#011(Stopped)
  192. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave vm_mckay#011(Stopped)
  193. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_sheppard:0#011(Slave -> Master vs001)
  194. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_sheppard:1#011(Stopped)
  195. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_sheppard-data:0#011(Slave -> Master vs001)
  196. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_sheppard-data:1#011(Stopped)
  197. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_sheppard#011(vs001)
  198. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_sheppard#011(vs001)
  199. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_tealc:0#011(Stopped)
  200. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_tealc:1#011(Slave -> Master vs001)
  201. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_tealc-data:0#011(Stopped)
  202. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_tealc-data:1#011(Slave -> Master vs001)
  203. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_tealc#011(vs001)
  204. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start fs_tealc-data#011(vs001)
  205. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Start vm_tealc#011(vs001)
  206. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_cinder-SSD1:0#011(Stopped)
  207. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_cinder-SSD1:1#011(Stopped)
  208. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave vg_cinder-SSD1#011(Stopped)
  209. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave ip_cinder-SSD1#011(Stopped)
  210. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave cinder_cinder-SSD1#011(Stopped)
  211. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave drbd_cinder-HDD1:0#011(Stopped)
  212. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Promote drbd_cinder-HDD1:1#011(Slave -> Master vs001)
  213. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave vg_cinder-HDD1#011(Stopped)
  214. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave ip_cinder-HDD1#011(Stopped)
  215. Nov 27 01:18:41 vs001 pengine: [1015]: notice: LogActions: Leave cinder_cinder-HDD1#011(Stopped)
  216. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vg_cinder-HDD1 (1385477804)
  217. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc:1 (10000)
  218. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_cinder-HDD1:1 (10000)
  219. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_hammond:1 (10000)
  220. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_rush:1 (10000)
  221. Nov 27 01:18:41 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard:0 (10000)
  222. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  223. Nov 27 01:18:41 vs001 crmd: [1016]: info: unpack_graph: Unpacked transition 0: 152 actions in 152 synapses
  224. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1385511521-83) derived from /var/lib/pengine/pe-input-297.bz2
  225. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 8: cancel drbd_rush:1_monitor_31000 on vs001 (local)
  226. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[91] on drbd_rush:1 for client 1016, its parameters: drbd_resource=[rush] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_rush:1 ] CRM_meta_notify_inactive_resource=[drbd_rush:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone_node_max=[1] cancelled
  227. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_rush:1_monitor_31000 from 8:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-85
  228. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-85 from vs001
  229. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_rush:1_monitor_31000 (8) confirmed on vs001 (rc=0)
  230. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
  231. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 12: cancel drbd_ronon:1_monitor_31000 on vs001 (local)
  232. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[92] on drbd_ronon:1 for client 1016, its parameters: drbd_resource=[ronon-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_ronon:1 ] CRM_meta_notify_inactive_resource=[drbd_ronon:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone_node cancelled
  233. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_monitor_31000 from 12:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-87
  234. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-87 from vs001
  235. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_ronon:1_monitor_31000 (12) confirmed on vs001 (rc=0)
  236. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 72 fired and confirmed
  237. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 1: cancel drbd_hammond:1_monitor_31000 on vs001 (local)
  238. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[93] on drbd_hammond:1 for client 1016, its parameters: drbd_resource=[hammond-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_hammond:1 ] CRM_meta_notify_inactive_resource=[drbd_hammond:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_cl cancelled
  239. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_hammond:1_monitor_31000 from 1:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-89
  240. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-89 from vs001
  241. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_hammond:1_monitor_31000 (1) confirmed on vs001 (rc=0)
  242. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 108 fired and confirmed
  243. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 6: cancel drbd_carter:1_monitor_31000 on vs001 (local)
  244. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[94] on drbd_carter:1 for client 1016, its parameters: drbd_resource=[carter-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_carter:1 ] CRM_meta_notify_inactive_resource=[drbd_carter:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone_ cancelled
  245. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_monitor_31000 from 6:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-91
  246. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-91 from vs001
  247. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_carter:1_monitor_31000 (6) confirmed on vs001 (rc=0)
  248. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 144 fired and confirmed
  249. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 5: cancel drbd_carter-data:1_monitor_31000 on vs001 (local)
  250. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[107] on drbd_carter-data:1 for client 1016, its parameters: drbd_resource=[carter-data] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_carter-data:1 ] CRM_meta_notify_inactive_resource=[drbd_carter-data:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] cancelled
  251. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter-data:1_monitor_31000 from 5:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-93
  252. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-93 from vs001
  253. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_carter-data:1_monitor_31000 (5) confirmed on vs001 (rc=0)
  254. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 173 fired and confirmed
  255. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 7: cancel drbd_oneill:1_monitor_31000 on vs001 (local)
  256. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[108] on drbd_oneill:1 for client 1016, its parameters: drbd_resource=[oneill-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_oneill:1 ] CRM_meta_notify_inactive_resource=[drbd_oneill:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone cancelled
  257. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_oneill:1_monitor_31000 from 7:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-95
  258. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-95 from vs001
  259. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_oneill:1_monitor_31000 (7) confirmed on vs001 (rc=0)
  260. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 209 fired and confirmed
  261. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 2: cancel drbd_backup:0_monitor_31000 on vs001 (local)
  262. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[109] on drbd_backup:0 for client 1016, its parameters: drbd_resource=[backup-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_backup:0 ] CRM_meta_notify_inactive_resource=[drbd_backup:0 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone cancelled
  263. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_monitor_31000 from 2:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-97
  264. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-97 from vs001
  265. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_backup:0_monitor_31000 (2) confirmed on vs001 (rc=0)
  266. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 245 fired and confirmed
  267. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 13: cancel drbd_backup-data:1_monitor_31000 on vs001 (local)
  268. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[110] on drbd_backup-data:1 for client 1016, its parameters: drbd_resource=[backup-data] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_backup-data:1 ] CRM_meta_notify_inactive_resource=[drbd_backup-data:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] cancelled
  269. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_monitor_31000 from 13:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-99
  270. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-99 from vs001
  271. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_backup-data:1_monitor_31000 (13) confirmed on vs001 (rc=0)
  272. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 274 fired and confirmed
  273. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 11: cancel drbd_sheppard:0_monitor_31000 on vs001 (local)
  274. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[95] on drbd_sheppard:0 for client 1016, its parameters: drbd_resource=[sheppard-root] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_sheppard:0 ] CRM_meta_notify_inactive_resource=[drbd_sheppard:0 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_met cancelled
  275. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_monitor_31000 from 11:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-101
  276. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-101 from vs001
  277. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_sheppard:0_monitor_31000 (11) confirmed on vs001 (rc=0)
  278. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 339 fired and confirmed
  279. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 3: cancel drbd_sheppard-data:0_monitor_31000 on vs001 (local)
  280. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[96] on drbd_sheppard-data:0 for client 1016, its parameters: drbd_resource=[sheppard-data] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_sheppard-data:0 ] CRM_meta_notify_inactive_resource=[drbd_sheppard-data:0 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[ cancelled
  281. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_monitor_31000 from 3:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-103
  282. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-103 from vs001
  283. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_sheppard-data:0_monitor_31000 (3) confirmed on vs001 (rc=0)
  284. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 368 fired and confirmed
  285. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 4: cancel drbd_tealc:1_monitor_31000 on vs001 (local)
  286. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[97] on drbd_tealc:1 for client 1016, its parameters: drbd_resource=[tealc-root2] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_tealc:1 ] CRM_meta_notify_inactive_resource=[drbd_tealc:1 drbd_tealc:2 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_m cancelled
  287. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_monitor_31000 from 4:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-105
  288. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-105 from vs001
  289. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_tealc:1_monitor_31000 (4) confirmed on vs001 (rc=0)
  290. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 404 fired and confirmed
  291. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 9: cancel drbd_tealc-data:1_monitor_31000 on vs001 (local)
  292. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[98] on drbd_tealc-data:1 for client 1016, its parameters: drbd_resource=[tealc-data2] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_tealc-data:1 ] CRM_meta_notify_inactive_resource=[drbd_tealc-data:1 drbd_tealc-data:2 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_ cancelled
  293. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_monitor_31000 from 9:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-107
  294. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-107 from vs001
  295. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_tealc-data:1_monitor_31000 (9) confirmed on vs001 (rc=0)
  296. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 433 fired and confirmed
  297. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 10: cancel drbd_cinder-HDD1:1_monitor_31000 on vs001 (local)
  298. Nov 27 01:18:41 vs001 lrmd: [1013]: info: cancel_op: operation monitor[179] on drbd_cinder-HDD1:1 for client 1016, its parameters: drbd_resource=[cinder-HDD1] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_cinder-HDD1:0 drbd_cinder-HDD1:1 ] CRM_meta_notify_inactive_resource=[drbd_cinder-HDD1:0 drbd_cinder-HDD1:1 ] CRM_meta_notify_master_uname=[ ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM cancelled
  299. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_monitor_31000 from 10:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-109
  300. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-109 from vs001
  301. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_cinder-HDD1:1_monitor_31000 (10) confirmed on vs001 (rc=0)
  302. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 498 fired and confirmed
  303. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_monitor_31000 (call=91, status=1, cib-update=0, confirmed=true) Cancelled
  304. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_monitor_31000 (call=92, status=1, cib-update=0, confirmed=true) Cancelled
  305. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_monitor_31000 (call=93, status=1, cib-update=0, confirmed=true) Cancelled
  306. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_monitor_31000 (call=94, status=1, cib-update=0, confirmed=true) Cancelled
  307. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_monitor_31000 (call=107, status=1, cib-update=0, confirmed=true) Cancelled
  308. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_monitor_31000 (call=108, status=1, cib-update=0, confirmed=true) Cancelled
  309. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_monitor_31000 (call=109, status=1, cib-update=0, confirmed=true) Cancelled
  310. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_monitor_31000 (call=110, status=1, cib-update=0, confirmed=true) Cancelled
  311. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_monitor_31000 (call=95, status=1, cib-update=0, confirmed=true) Cancelled
  312. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_monitor_31000 (call=96, status=1, cib-update=0, confirmed=true) Cancelled
  313. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_monitor_31000 (call=97, status=1, cib-update=0, confirmed=true) Cancelled
  314. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_monitor_31000 (call=98, status=1, cib-update=0, confirmed=true) Cancelled
  315. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_monitor_31000 (call=179, status=1, cib-update=0, confirmed=true) Cancelled
  316. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 583: notify drbd_rush:1_pre_notify_promote_0 on vs001 (local)
  317. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=583:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_rush:1_notify_0 )
  318. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 notify[180] (pid 31004)
  319. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 591: notify drbd_ronon:1_pre_notify_promote_0 on vs001 (local)
  320. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=591:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_ronon:1_notify_0 )
  321. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 notify[181] (pid 31005)
  322. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 599: notify drbd_hammond:1_pre_notify_promote_0 on vs001 (local)
  323. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=599:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_hammond:1_notify_0 )
  324. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 notify[182] (pid 31006)
  325. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 607: notify drbd_carter:1_pre_notify_promote_0 on vs001 (local)
  326. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=607:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_carter:1_notify_0 )
  327. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 notify[183] (pid 31009)
  328. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 615: notify drbd_carter-data:1_pre_notify_promote_0 on vs001 (local)
  329. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=615:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_carter-data:1_notify_0 )
  330. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 623: notify drbd_oneill:1_pre_notify_promote_0 on vs001 (local)
  331. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=623:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_oneill:1_notify_0 )
  332. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 631: notify drbd_backup:0_pre_notify_promote_0 on vs001 (local)
  333. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=631:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_backup:0_notify_0 )
  334. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 639: notify drbd_backup-data:1_pre_notify_promote_0 on vs001 (local)
  335. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=639:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_backup-data:1_notify_0 )
  336. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 647: notify drbd_sheppard:0_pre_notify_promote_0 on vs001 (local)
  337. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=647:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_sheppard:0_notify_0 )
  338. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 655: notify drbd_sheppard-data:0_pre_notify_promote_0 on vs001 (local)
  339. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=655:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_sheppard-data:0_notify_0 )
  340. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 663: notify drbd_tealc:1_pre_notify_promote_0 on vs001 (local)
  341. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=663:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_tealc:1_notify_0 )
  342. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 671: notify drbd_tealc-data:1_pre_notify_promote_0 on vs001 (local)
  343. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=671:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_tealc-data:1_notify_0 )
  344. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 679: notify drbd_cinder-HDD1:1_pre_notify_promote_0 on vs001 (local)
  345. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=679:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_cinder-HDD1:1_notify_0 )
  346. Nov 27 01:18:41 vs001 lrmd: [1013]: info: operation notify[180] on drbd_rush:1 for client 1016: pid 31004 exited with return code 0
  347. Nov 27 01:18:41 vs001 lrmd: [1013]: info: operation notify[183] on drbd_carter:1 for client 1016: pid 31009 exited with return code 0
  348. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_rush:1_notify_0 from 583:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-123
  349. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-123 from vs001
  350. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_rush:1_notify_0 (583) confirmed on vs001 (rc=0)
  351. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_notify_0 (call=180, rc=0, cib-update=0, confirmed=true) ok
  352. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_notify_0 from 607:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-124
  353. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-124 from vs001
  354. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_carter:1_notify_0 (607) confirmed on vs001 (rc=0)
  355. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_notify_0 (call=183, rc=0, cib-update=0, confirmed=true) ok
  356. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
  357. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
  358. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 145 fired and confirmed
  359. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 142 fired and confirmed
  360. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 20: promote drbd_rush:1_promote_0 on vs001 (local)
  361. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=20:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_rush:1_promote_0 )
  362. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 promote[193] (pid 31088)
  363. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 128: promote drbd_carter:1_promote_0 on vs001 (local)
  364. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=128:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_carter:1_promote_0 )
  365. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 promote[194] (pid 31089)
  366. Nov 27 01:18:41 vs001 lrmd: [1013]: info: operation notify[181] on drbd_ronon:1 for client 1016: pid 31005 exited with return code 0
  367. Nov 27 01:18:41 vs001 lrmd: [1013]: info: operation notify[182] on drbd_hammond:1 for client 1016: pid 31006 exited with return code 0
  368. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_notify_0 from 591:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-127
  369. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-127 from vs001
  370. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_ronon:1_notify_0 (591) confirmed on vs001 (rc=0)
  371. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_notify_0 (call=181, rc=0, cib-update=0, confirmed=true) ok
  372. Nov 27 01:18:41 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_hammond:1_notify_0 from 599:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511521-128
  373. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511521-128 from vs001
  374. Nov 27 01:18:41 vs001 crmd: [1016]: info: match_graph_event: Action drbd_hammond:1_notify_0 (599) confirmed on vs001 (rc=0)
  375. Nov 27 01:18:41 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_notify_0 (call=182, rc=0, cib-update=0, confirmed=true) ok
  376. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 73 fired and confirmed
  377. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 70 fired and confirmed
  378. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 109 fired and confirmed
  379. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 106 fired and confirmed
  380. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 56: promote drbd_ronon:1_promote_0 on vs001 (local)
  381. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=56:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_ronon:1_promote_0 )
  382. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 promote[195] (pid 31104)
  383. Nov 27 01:18:41 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 92: promote drbd_hammond:1_promote_0 on vs001 (local)
  384. Nov 27 01:18:41 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=92:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_hammond:1_promote_0 )
  385. Nov 27 01:18:41 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 promote[196] (pid 31105)
  386. Nov 27 01:18:41 vs001 pengine: [1015]: notice: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/pengine/pe-input-297.bz2
  387. Nov 27 01:18:42 vs001 kernel: [44153.813837] block drbd4: role( Secondary -> Primary )
  388. Nov 27 01:18:42 vs001 kernel: [44153.861428] block drbd27: PingAck did not arrive in time.
  389. Nov 27 01:18:42 vs001 kernel: [44153.861439] block drbd27: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  390. Nov 27 01:18:42 vs001 kernel: [44153.955385] block drbd27: asender terminated
  391. Nov 27 01:18:42 vs001 kernel: [44153.955394] block drbd27: Terminating drbd27_asender
  392. Nov 27 01:18:42 vs001 kernel: [44153.955421] block drbd4: new current UUID 8C9AEB8D246A26CD:2205F7F942B19D7C:A6AA3C18C50817EA:A6A93C18C50817EB
  393. Nov 27 01:18:42 vs001 kernel: [44153.955530] block drbd27: Connection closed
  394. Nov 27 01:18:42 vs001 kernel: [44153.955539] block drbd27: conn( NetworkFailure -> Unconnected )
  395. Nov 27 01:18:42 vs001 kernel: [44153.955546] block drbd27: receiver terminated
  396. Nov 27 01:18:42 vs001 kernel: [44153.955550] block drbd27: Restarting drbd27_receiver
  397. Nov 27 01:18:42 vs001 kernel: [44153.955554] block drbd27: receiver (re)started
  398. Nov 27 01:18:42 vs001 kernel: [44153.955561] block drbd27: conn( Unconnected -> WFConnection )
  399. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_ronon:1:promote:stdout)
  400. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation promote[195] on drbd_ronon:1 for client 1016: pid 31104 exited with return code 0
  401. Nov 27 01:18:42 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_promote_0 (call=195, rc=0, cib-update=157, confirmed=true) ok
  402. Nov 27 01:18:42 vs001 crmd: [1016]: info: match_graph_event: Action drbd_ronon:1_promote_0 (56) confirmed on vs001 (rc=0)
  403. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 71 fired and confirmed
  404. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 74 fired and confirmed
  405. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 592: notify drbd_ronon:1_post_notify_promote_0 on vs001 (local)
  406. Nov 27 01:18:42 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=592:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_ronon:1_notify_0 )
  407. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 notify[197] (pid 31206)
  408. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_ronon:1:notify:stdout)
  409. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[197] on drbd_ronon:1 for client 1016: pid 31206 exited with return code 0
  410. Nov 27 01:18:42 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_notify_0 from 592:0:0:2a9d944e-4153-417a-a3c1-e9a5c461707d: lrm_invoke-lrmd-1385511522-132
  411. Nov 27 01:18:42 vs001 crmd: [1016]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1385511522-132 from vs001
  412. Nov 27 01:18:42 vs001 crmd: [1016]: info: match_graph_event: Action drbd_ronon:1_notify_0 (592) confirmed on vs001 (rc=0)
  413. Nov 27 01:18:42 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_notify_0 (call=197, rc=0, cib-update=0, confirmed=true) ok
  414. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 75 fired and confirmed
  415. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_pseudo_action: Pseudo action 85 fired and confirmed
  416. Nov 27 01:18:42 vs001 crmd: [1016]: info: te_rsc_command: Initiating action 57: monitor drbd_ronon:1_monitor_29000 on vs001 (local)
  417. Nov 27 01:18:42 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=57:0:8:2a9d944e-4153-417a-a3c1-e9a5c461707d op=drbd_ronon:1_monitor_29000 )
  418. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 monitor[198] (pid 31234)
  419. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation monitor[198] on drbd_ronon:1 for client 1016: pid 31234 exited with return code 8
  420. Nov 27 01:18:42 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_monitor_29000 (call=198, rc=8, cib-update=158, confirmed=false) master
  421. Nov 27 01:18:42 vs001 crmd: [1016]: info: match_graph_event: Action drbd_ronon:1_monitor_29000 (57) confirmed on vs001 (rc=0)
  422. Nov 27 01:18:42 vs001 kernel: [44154.293456] block drbd0: PingAck did not arrive in time.
  423. Nov 27 01:18:42 vs001 kernel: [44154.293496] block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  424. Nov 27 01:18:42 vs001 kernel: [44154.305411] block drbd3: PingAck did not arrive in time.
  425. Nov 27 01:18:42 vs001 kernel: [44154.305449] block drbd3: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  426. Nov 27 01:18:42 vs001 kernel: [44154.309417] block drbd2: PingAck did not arrive in time.
  427. Nov 27 01:18:42 vs001 kernel: [44154.309459] block drbd2: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  428. Nov 27 01:18:42 vs001 kernel: [44154.335699] block drbd0: asender terminated
  429. Nov 27 01:18:42 vs001 kernel: [44154.335705] block drbd0: Terminating drbd0_asender
  430. Nov 27 01:18:42 vs001 kernel: [44154.335888] block drbd0: Connection closed
  431. Nov 27 01:18:42 vs001 kernel: [44154.335896] block drbd0: conn( NetworkFailure -> Unconnected )
  432. Nov 27 01:18:42 vs001 kernel: [44154.335902] block drbd0: receiver terminated
  433. Nov 27 01:18:42 vs001 kernel: [44154.335904] block drbd0: Restarting drbd0_receiver
  434. Nov 27 01:18:42 vs001 kernel: [44154.335906] block drbd0: receiver (re)started
  435. Nov 27 01:18:42 vs001 kernel: [44154.393413] block drbd0: role( Secondary -> Primary )
  436. Nov 27 01:18:42 vs001 kernel: [44154.402186] block drbd2: asender terminated
  437. Nov 27 01:18:42 vs001 kernel: [44154.402192] block drbd2: Terminating drbd2_asender
  438. Nov 27 01:18:42 vs001 kernel: [44154.402220] block drbd3: asender terminated
  439. Nov 27 01:18:42 vs001 kernel: [44154.402227] block drbd3: Terminating drbd3_asender
  440. Nov 27 01:18:42 vs001 kernel: [44154.402295] block drbd2: Connection closed
  441. Nov 27 01:18:42 vs001 kernel: [44154.402301] block drbd2: conn( NetworkFailure -> Unconnected )
  442. Nov 27 01:18:42 vs001 kernel: [44154.402305] block drbd2: receiver terminated
  443. Nov 27 01:18:42 vs001 kernel: [44154.402308] block drbd2: Restarting drbd2_receiver
  444. Nov 27 01:18:42 vs001 kernel: [44154.402311] block drbd2: receiver (re)started
  445. Nov 27 01:18:42 vs001 kernel: [44154.402352] block drbd3: Connection closed
  446. Nov 27 01:18:42 vs001 kernel: [44154.402356] block drbd3: conn( NetworkFailure -> Unconnected )
  447. Nov 27 01:18:42 vs001 kernel: [44154.402360] block drbd3: receiver terminated
  448. Nov 27 01:18:42 vs001 kernel: [44154.402362] block drbd3: Restarting drbd3_receiver
  449. Nov 27 01:18:42 vs001 kernel: [44154.402364] block drbd3: receiver (re)started
  450. Nov 27 01:18:42 vs001 kernel: [44154.405414] block drbd3: role( Secondary -> Primary )
  451. Nov 27 01:18:42 vs001 kernel: [44154.409420] block drbd2: role( Secondary -> Primary )
  452. Nov 27 01:18:42 vs001 kernel: [44154.437347] block drbd0: new current UUID F00AE33D316E99AF:0C721E70E838AA92:B0BDE6667D9CF606:B0BCE6667D9CF607
  453. Nov 27 01:18:42 vs001 kernel: [44154.501413] block drbd30: PingAck did not arrive in time.
  454. Nov 27 01:18:42 vs001 kernel: [44154.502232] block drbd30: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  455. Nov 27 01:18:42 vs001 kernel: [44154.521366] block drbd2: new current UUID B6D48CC19119A6B7:0CEDB5E1042B389C:AF20123A68A2C4AC:AF1F123A68A2C4AD
  456. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_rush:1:promote:stdout)
  457. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation promote[193] on drbd_rush:1 for client 1016: pid 31088 exited with return code 0
  458. Nov 27 01:18:42 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_promote_0 (call=193, rc=0, cib-update=159, confirmed=true) ok
  459. Nov 27 01:18:42 vs001 kernel: [44154.521489] block drbd0: conn( Unconnected -> WFConnection )
  460. Nov 27 01:18:42 vs001 kernel: [44154.521609] block drbd3: new current UUID 20548E8051DBC995:C35416C6C7C7F778:6D3197FD0EA3C69A:6D3097FD0EA3C69B
  461. Nov 27 01:18:42 vs001 crmd: [1016]: info: match_graph_event: Action drbd_rush:1_promote_0 (20) confirmed on vs001 (rc=0)
  462. Nov 27 01:18:42 vs001 rsyslogd-2177: imuxsock begins to drop messages from pid 1016 due to rate-limiting
  463. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 notify[199] (pid 31276)
  464. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_rush:1:notify:stdout)
  465. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[199] on drbd_rush:1 for client 1016: pid 31276 exited with return code 0
  466. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 monitor[200] (pid 31304)
  467. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation monitor[200] on drbd_rush:1 for client 1016: pid 31304 exited with return code 8
  468. Nov 27 01:18:42 vs001 kernel: [44154.578360] block drbd30: asender terminated
  469. Nov 27 01:18:42 vs001 kernel: [44154.578366] block drbd30: Terminating drbd30_asender
  470. Nov 27 01:18:42 vs001 kernel: [44154.578462] block drbd30: Connection closed
  471. Nov 27 01:18:42 vs001 kernel: [44154.578468] block drbd30: conn( NetworkFailure -> Unconnected )
  472. Nov 27 01:18:42 vs001 kernel: [44154.578471] block drbd30: receiver terminated
  473. Nov 27 01:18:42 vs001 kernel: [44154.578472] block drbd30: Restarting drbd30_receiver
  474. Nov 27 01:18:42 vs001 kernel: [44154.578474] block drbd30: receiver (re)started
  475. Nov 27 01:18:42 vs001 kernel: [44154.578477] block drbd30: conn( Unconnected -> WFConnection )
  476. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_hammond:1:promote:stdout)
  477. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_carter:1:promote:stdout)
  478. Nov 27 01:18:42 vs001 kernel: [44154.650928] block drbd3: conn( Unconnected -> WFConnection )
  479. Nov 27 01:18:42 vs001 kernel: [44154.651011] block drbd2: conn( Unconnected -> WFConnection )
  480. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation promote[194] on drbd_carter:1 for client 1016: pid 31089 exited with return code 0
  481. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation promote[196] on drbd_hammond:1 for client 1016: pid 31105 exited with return code 0
  482. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 notify[201] (pid 31334)
  483. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 notify[202] (pid 31339)
  484. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_carter:1:notify:stdout)
  485. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[201] on drbd_carter:1 for client 1016: pid 31334 exited with return code 0
  486. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 monitor[203] (pid 31384)
  487. Nov 27 01:18:42 vs001 lrmd: [1013]: info: RA output: (drbd_hammond:1:notify:stdout)
  488. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[202] on drbd_hammond:1 for client 1016: pid 31339 exited with return code 0
  489. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 monitor[204] (pid 31395)
  490. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 notify[184] (pid 31424)
  491. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 notify[185] (pid 31427)
  492. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation monitor[203] on drbd_carter:1 for client 1016: pid 31384 exited with return code 8
  493. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation monitor[204] on drbd_hammond:1 for client 1016: pid 31395 exited with return code 8
  494. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[184] on drbd_carter-data:1 for client 1016: pid 31424 exited with return code 0
  495. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 promote[205] (pid 31495)
  496. Nov 27 01:18:42 vs001 lrmd: [1013]: info: operation notify[185] on drbd_oneill:1 for client 1016: pid 31427 exited with return code 0
  497. Nov 27 01:18:42 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 promote[206] (pid 31498)
  498. Nov 27 01:18:43 vs001 kernel: [44154.965405] block drbd31: PingAck did not arrive in time.
  499. Nov 27 01:18:43 vs001 kernel: [44154.966225] block drbd31: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  500. Nov 27 01:18:43 vs001 kernel: [44155.014197] block drbd31: asender terminated
  501. Nov 27 01:18:43 vs001 kernel: [44155.014203] block drbd31: Terminating drbd31_asender
  502. Nov 27 01:18:43 vs001 kernel: [44155.014381] block drbd31: Connection closed
  503. Nov 27 01:18:43 vs001 kernel: [44155.014389] block drbd31: conn( NetworkFailure -> Unconnected )
  504. Nov 27 01:18:43 vs001 kernel: [44155.014395] block drbd31: receiver terminated
  505. Nov 27 01:18:43 vs001 kernel: [44155.014397] block drbd31: Restarting drbd31_receiver
  506. Nov 27 01:18:43 vs001 kernel: [44155.014400] block drbd31: receiver (re)started
  507. Nov 27 01:18:43 vs001 kernel: [44155.014404] block drbd31: conn( Unconnected -> WFConnection )
  508. Nov 27 01:18:43 vs001 kernel: [44155.273392] block drbd7: PingAck did not arrive in time.
  509. Nov 27 01:18:43 vs001 kernel: [44155.273399] block drbd1: PingAck did not arrive in time.
  510. Nov 27 01:18:43 vs001 kernel: [44155.273410] block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  511. Nov 27 01:18:43 vs001 kernel: [44155.275040] block drbd7: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  512. Nov 27 01:18:43 vs001 kernel: [44155.359266] block drbd7: asender terminated
  513. Nov 27 01:18:43 vs001 kernel: [44155.359272] block drbd7: Terminating drbd7_asender
  514. Nov 27 01:18:43 vs001 kernel: [44155.359300] block drbd1: asender terminated
  515. Nov 27 01:18:43 vs001 kernel: [44155.359307] block drbd1: Terminating drbd1_asender
  516. Nov 27 01:18:43 vs001 kernel: [44155.359338] block drbd7: Connection closed
  517. Nov 27 01:18:43 vs001 kernel: [44155.359345] block drbd7: conn( NetworkFailure -> Unconnected )
  518. Nov 27 01:18:43 vs001 kernel: [44155.359349] block drbd7: receiver terminated
  519. Nov 27 01:18:43 vs001 kernel: [44155.359353] block drbd7: Restarting drbd7_receiver
  520. Nov 27 01:18:43 vs001 kernel: [44155.359356] block drbd7: receiver (re)started
  521. Nov 27 01:18:43 vs001 kernel: [44155.359378] block drbd1: Connection closed
  522. Nov 27 01:18:43 vs001 kernel: [44155.359384] block drbd1: conn( NetworkFailure -> Unconnected )
  523. Nov 27 01:18:43 vs001 kernel: [44155.359388] block drbd1: receiver terminated
  524. Nov 27 01:18:43 vs001 kernel: [44155.359390] block drbd1: Restarting drbd1_receiver
  525. Nov 27 01:18:43 vs001 kernel: [44155.359393] block drbd1: receiver (re)started
  526. Nov 27 01:18:43 vs001 kernel: [44155.373394] block drbd1: role( Secondary -> Primary )
  527. Nov 27 01:18:43 vs001 kernel: [44155.373414] block drbd7: role( Secondary -> Primary )
  528. Nov 27 01:18:43 vs001 kernel: [44155.430456] block drbd7: new current UUID 5A996DCBBB8B8D65:9F4F613EF03DCD86:7C58AB812E3670D6:7C57AB812E3670D7
  529. Nov 27 01:18:43 vs001 kernel: [44155.430500] block drbd1: new current UUID 5B155798912405BD:799281900832AAC6:AFD08B5063E56D3E:AFCF8B5063E56D3F
  530. Nov 27 01:18:43 vs001 kernel: [44155.492632] block drbd1: conn( Unconnected -> WFConnection )
  531. Nov 27 01:18:43 vs001 kernel: [44155.492811] block drbd7: conn( Unconnected -> WFConnection )
  532. Nov 27 01:18:43 vs001 lrmd: [1013]: info: RA output: (drbd_carter-data:1:promote:stdout)
  533. Nov 27 01:18:43 vs001 lrmd: [1013]: info: RA output: (drbd_oneill:1:promote:stdout)
  534. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation promote[205] on drbd_carter-data:1 for client 1016: pid 31495 exited with return code 0
  535. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation promote[206] on drbd_oneill:1 for client 1016: pid 31498 exited with return code 0
  536. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 notify[207] (pid 31583)
  537. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 notify[208] (pid 31588)
  538. Nov 27 01:18:43 vs001 lrmd: [1013]: info: RA output: (drbd_carter-data:1:notify:stdout)
  539. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[207] on drbd_carter-data:1 for client 1016: pid 31583 exited with return code 0
  540. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 monitor[209] (pid 31633)
  541. Nov 27 01:18:43 vs001 lrmd: [1013]: info: RA output: (drbd_oneill:1:notify:stdout)
  542. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[208] on drbd_oneill:1 for client 1016: pid 31588 exited with return code 0
  543. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 monitor[210] (pid 31644)
  544. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation monitor[209] on drbd_carter-data:1 for client 1016: pid 31633 exited with return code 8
  545. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation monitor[210] on drbd_oneill:1 for client 1016: pid 31644 exited with return code 8
  546. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[186] (pid 31698)
  547. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[187] (pid 31708)
  548. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[188] (pid 31712)
  549. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[189] (pid 31713)
  550. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[186] on drbd_backup:0 for client 1016: pid 31698 exited with return code 0
  551. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 promote[211] (pid 31737)
  552. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[187] on drbd_backup-data:1 for client 1016: pid 31708 exited with return code 0
  553. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[188] on drbd_sheppard:0 for client 1016: pid 31712 exited with return code 0
  554. Nov 27 01:18:43 vs001 lrmd: [1013]: info: operation notify[189] on drbd_sheppard-data:0 for client 1016: pid 31713 exited with return code 0
  555. Nov 27 01:18:43 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 promote[212] (pid 31802)
  556. Nov 27 01:18:43 vs001 kernel: [44155.768807] block drbd31: role( Secondary -> Primary )
  557. Nov 27 01:18:43 vs001 kernel: [44155.779540] block drbd21: role( Secondary -> Primary )
  558. Nov 27 01:18:44 vs001 kernel: [44155.938347] block drbd21: new current UUID 0229ABADF49E173B:62FC58A2B4B5A44C:94E62243BFDE1B1E:94E52243BFDE1B1F
  559. Nov 27 01:18:44 vs001 kernel: [44155.938389] block drbd31: new current UUID F970ACEF0F86767D:AEA81E5CB7BF6942:4BBB731F08D87EBA:4BBA731F08D87EBB
  560. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_backup:0:promote:stdout)
  561. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard:0:promote:stdout)
  562. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation promote[212] on drbd_sheppard:0 for client 1016: pid 31802 exited with return code 0
  563. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation promote[211] on drbd_backup:0 for client 1016: pid 31737 exited with return code 0
  564. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[213] (pid 31863)
  565. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[214] (pid 31869)
  566. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard:0:notify:stdout)
  567. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[213] on drbd_sheppard:0 for client 1016: pid 31863 exited with return code 0
  568. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 monitor[215] (pid 31913)
  569. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 promote[216] (pid 31915)
  570. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_backup:0:notify:stdout)
  571. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[214] on drbd_backup:0 for client 1016: pid 31869 exited with return code 0
  572. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 monitor[217] (pid 31934)
  573. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 promote[218] (pid 31938)
  574. Nov 27 01:18:44 vs001 kernel: [44156.062250] block drbd27: role( Secondary -> Primary )
  575. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation monitor[215] on drbd_sheppard:0 for client 1016: pid 31913 exited with return code 8
  576. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation monitor[217] on drbd_backup:0 for client 1016: pid 31934 exited with return code 8
  577. Nov 27 01:18:44 vs001 kernel: [44156.134902] block drbd27: new current UUID 704C531570446E6F:3B73B12444735996:A2619AFEB006CDA8:A2609AFEB006CDA9
  578. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard-data:0:promote:stdout)
  579. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation promote[216] on drbd_sheppard-data:0 for client 1016: pid 31915 exited with return code 0
  580. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[219] (pid 32036)
  581. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard-data:0:notify:stdout)
  582. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[219] on drbd_sheppard-data:0 for client 1016: pid 32036 exited with return code 0
  583. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 monitor[220] (pid 32064)
  584. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation monitor[220] on drbd_sheppard-data:0 for client 1016: pid 32064 exited with return code 8
  585. Nov 27 01:18:44 vs001 kernel: [44156.517393] block drbd20: PingAck did not arrive in time.
  586. Nov 27 01:18:44 vs001 kernel: [44156.518172] block drbd20: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  587. Nov 27 01:18:44 vs001 kernel: [44156.570857] block drbd20: asender terminated
  588. Nov 27 01:18:44 vs001 kernel: [44156.570863] block drbd20: Terminating drbd20_asender
  589. Nov 27 01:18:44 vs001 kernel: [44156.571036] block drbd20: Connection closed
  590. Nov 27 01:18:44 vs001 kernel: [44156.571043] block drbd20: conn( NetworkFailure -> Unconnected )
  591. Nov 27 01:18:44 vs001 kernel: [44156.571049] block drbd20: receiver terminated
  592. Nov 27 01:18:44 vs001 kernel: [44156.571051] block drbd20: Restarting drbd20_receiver
  593. Nov 27 01:18:44 vs001 kernel: [44156.571054] block drbd20: receiver (re)started
  594. Nov 27 01:18:44 vs001 kernel: [44156.571058] block drbd20: conn( Unconnected -> WFConnection )
  595. Nov 27 01:18:44 vs001 kernel: [44156.573362] block drbd32: PingAck did not arrive in time.
  596. Nov 27 01:18:44 vs001 kernel: [44156.574137] block drbd32: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
  597. Nov 27 01:18:44 vs001 kernel: [44156.606900] block drbd32: asender terminated
  598. Nov 27 01:18:44 vs001 kernel: [44156.606906] block drbd32: Terminating drbd32_asender
  599. Nov 27 01:18:44 vs001 kernel: [44156.607069] block drbd32: Connection closed
  600. Nov 27 01:18:44 vs001 kernel: [44156.607076] block drbd32: conn( NetworkFailure -> Unconnected )
  601. Nov 27 01:18:44 vs001 kernel: [44156.607082] block drbd32: receiver terminated
  602. Nov 27 01:18:44 vs001 kernel: [44156.607084] block drbd32: Restarting drbd32_receiver
  603. Nov 27 01:18:44 vs001 kernel: [44156.607087] block drbd32: receiver (re)started
  604. Nov 27 01:18:44 vs001 kernel: [44156.673351] block drbd32: role( Secondary -> Primary )
  605. Nov 27 01:18:44 vs001 kernel: [44156.695858] block drbd32: new current UUID 8AFAF16B682F0BC9:2ACC972D85254D92:0B73122907A5B5F0:0B72122907A5B5F1
  606. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_backup-data:1:promote:stdout)
  607. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[190] (pid 32116)
  608. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[191] (pid 32117)
  609. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[192] (pid 32118)
  610. Nov 27 01:18:44 vs001 kernel: [44156.718069] block drbd32: conn( Unconnected -> WFConnection )
  611. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation promote[218] on drbd_backup-data:1 for client 1016: pid 31938 exited with return code 0
  612. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[221] (pid 32129)
  613. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[190] on drbd_tealc:1 for client 1016: pid 32116 exited with return code 0
  614. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 promote[222] (pid 32202)
  615. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[192] on drbd_cinder-HDD1:1 for client 1016: pid 32118 exited with return code 0
  616. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[191] on drbd_tealc-data:1 for client 1016: pid 32117 exited with return code 0
  617. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 promote[223] (pid 32206)
  618. Nov 27 01:18:44 vs001 lrmd: [1013]: info: RA output: (drbd_backup-data:1:notify:stdout)
  619. Nov 27 01:18:44 vs001 lrmd: [1013]: info: operation notify[221] on drbd_backup-data:1 for client 1016: pid 32129 exited with return code 0
  620. Nov 27 01:18:44 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 monitor[224] (pid 32244)
  621. Nov 27 01:18:44 vs001 kernel: [44156.782690] block drbd20: role( Secondary -> Primary )
  622. Nov 27 01:18:44 vs001 kernel: [44156.784350] block drbd16: role( Secondary -> Primary )
  623. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation monitor[224] on drbd_backup-data:1 for client 1016: pid 32244 exited with return code 8
  624. Nov 27 01:18:45 vs001 kernel: [44156.870893] block drbd20: new current UUID 7AAA1A5F1D99C87F:6A7320496FAF9214:E52B26590621112E:E52A26590621112F
  625. Nov 27 01:18:45 vs001 kernel: [44156.870914] block drbd16: new current UUID FDC1951AA7D37BFB:A1978A43833EBAD4:0001000000000004:0000000000000004
  626. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:promote:stdout)
  627. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_tealc:1:promote:stdout)
  628. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation promote[222] on drbd_tealc:1 for client 1016: pid 32202 exited with return code 0
  629. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation promote[223] on drbd_cinder-HDD1:1 for client 1016: pid 32206 exited with return code 0
  630. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[225] (pid 32308)
  631. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[226] (pid 32316)
  632. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_tealc:1:notify:stdout)
  633. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation notify[225] on drbd_tealc:1 for client 1016: pid 32308 exited with return code 0
  634. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 monitor[227] (pid 32358)
  635. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 promote[228] (pid 32361)
  636. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:notify:stdout)
  637. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation notify[226] on drbd_cinder-HDD1:1 for client 1016: pid 32316 exited with return code 0
  638. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 monitor[229] (pid 32378)
  639. Nov 27 01:18:45 vs001 kernel: [44157.030560] block drbd30: role( Secondary -> Primary )
  640. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation monitor[227] on drbd_tealc:1 for client 1016: pid 32358 exited with return code 8
  641. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation monitor[229] on drbd_cinder-HDD1:1 for client 1016: pid 32378 exited with return code 8
  642. Nov 27 01:18:45 vs001 kernel: [44157.103845] block drbd30: new current UUID 43A9065AD827787B:BC2F5E64029D705A:148B2DB5AF7A3AF2:148A2DB5AF7A3AF3
  643. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_tealc-data:1:promote:stdout)
  644. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation promote[228] on drbd_tealc-data:1 for client 1016: pid 32361 exited with return code 0
  645. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[230] (pid 32468)
  646. Nov 27 01:18:45 vs001 lrmd: [1013]: info: RA output: (drbd_tealc-data:1:notify:stdout)
  647. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation notify[230] on drbd_tealc-data:1 for client 1016: pid 32468 exited with return code 0
  648. Nov 27 01:18:45 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 monitor[231] (pid 32499)
  649. Nov 27 01:18:45 vs001 lrmd: [1013]: info: operation monitor[231] on drbd_tealc-data:1 for client 1016: pid 32499 exited with return code 8
  650. Nov 27 01:20:01 vs001 CRON[1097]: (root) CMD ( ntpdate ntp.ubuntu.com)
  651. Nov 27 01:20:01 vs001 rsyslogd-2177: imuxsock lost 264 messages from pid 1016 due to rate-limiting
  652. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  653. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  654. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  655. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  656. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  657. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  658. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  659. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  660. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  661. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  662. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  663. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  664. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  665. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  666. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  667. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  668. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  669. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  670. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  671. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  672. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  673. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  674. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  675. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  676. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Timer popped (timeout=20000, abort_level=0, complete=true)
  677. Nov 27 01:20:01 vs001 crmd: [1016]: WARN: action_timer_callback: Ignoring timeout while not in transition
  678. Nov 27 01:20:10 vs001 CRON[1096]: (CRON) info (No MTA installed, discarding output)
  679. Nov 27 01:20:12 vs001 cib: [1012]: notice: ais_dispatch_message: Membership 756: quorum acquired
  680. Nov 27 01:20:12 vs001 crmd: [1016]: notice: ais_dispatch_message: Membership 756: quorum acquired
  681. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 756: memb=1, new=0, lost=0
  682. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: memb: vs001 603909211
  683. Nov 27 01:20:12 vs001 crmd: [1016]: info: ais_status_callback: status: vs002 is now member (was lost)
  684. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 756: memb=3, new=2, lost=0
  685. Nov 27 01:20:12 vs001 cib: [1012]: info: crm_update_peer: Node vs002: id=687795291 state=member (new) addr=r(0) ip(91.236.254.40) votes=1 born=560 seen=756 proc=00000000000000000000000000111312
  686. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: update_member: Node 687795291/vs002 is now: member
  687. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: NEW: vs002 687795291
  688. Nov 27 01:20:12 vs001 cib: [1012]: info: crm_update_peer: Node vs003: id=721349723 state=member (new) addr=r(0) ip(91.236.254.42) votes=1 born=148 seen=756 proc=00000000000000000000000000111312
  689. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: update_member: Node 721349723/vs003 is now: member
  690. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: NEW: vs003 721349723
  691. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: MEMB: vs001 603909211
  692. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: MEMB: vs002 687795291
  693. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: pcmk_peer_update: MEMB: vs003 721349723
  694. Nov 27 01:20:12 vs001 crmd: [1016]: WARN: match_down_event: No match for shutdown action on vs002
  695. Nov 27 01:20:12 vs001 corosync[3280]: [pcmk ] info: send_member_notification: Sending membership update 756 to 2 children
  696. Nov 27 01:20:12 vs001 corosync[3280]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  697. Nov 27 01:20:12 vs001 crmd: [1016]: info: crm_update_peer: Node vs002: id=687795291 state=member (new) addr=r(0) ip(91.236.254.40) votes=1 born=560 seen=756 proc=00000000000000000000000000111312
  698. Nov 27 01:20:12 vs001 crmd: [1016]: info: ais_status_callback: status: vs003 is now member (was lost)
  699. Nov 27 01:20:12 vs001 crmd: [1016]: WARN: match_down_event: No match for shutdown action on vs003
  700. Nov 27 01:20:12 vs001 crmd: [1016]: info: crm_update_peer: Node vs003: id=721349723 state=member (new) addr=r(0) ip(91.236.254.42) votes=1 born=148 seen=756 proc=00000000000000000000000000111312
  701. Nov 27 01:20:12 vs001 crmd: [1016]: info: crm_update_quorum: Updating quorum status to true (call=189)
  702. Nov 27 01:20:12 vs001 corosync[3280]: [CPG ] chosen downlist: sender r(0) ip(91.236.254.40) ; members(old:2 left:0)
  703. Nov 27 01:20:12 vs001 corosync[3280]: [MAIN ] Completed service synchronization, ready to provide service.
  704. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vs002']/lrm (origin=local/crmd/183, version=0.629.356): ok (rc=0)
  705. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vs002']/transient_attributes (origin=local/crmd/184, version=0.629.357): ok (rc=0)
  706. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vs003']/lrm (origin=local/crmd/185, version=0.629.358): ok (rc=0)
  707. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vs003']/transient_attributes (origin=local/crmd/186, version=0.629.359): ok (rc=0)
  708. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/187, version=0.629.360): ok (rc=0)
  709. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_diff: Diff 0.629.477 -> 0.629.478 not applied to 0.629.360: current "num_updates" is less than required
  710. Nov 27 01:20:12 vs001 cib: [1012]: WARN: cib_server_process_diff: Not requesting full refresh in R/W mode
  711. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_diff: Diff 0.629.478 -> 0.629.479 not applied to 0.629.360: current "num_updates" is less than required
  712. Nov 27 01:20:12 vs001 cib: [1012]: WARN: cib_server_process_diff: Not requesting full refresh in R/W mode
  713. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_diff: Diff 0.629.479 -> 0.629.480 not applied to 0.629.360: current "num_updates" is less than required
  714. Nov 27 01:20:12 vs001 cib: [1012]: WARN: cib_server_process_diff: Not requesting full refresh in R/W mode
  715. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_diff: Diff 0.629.480 -> 0.629.481 not applied to 0.629.360: current "num_updates" is less than required
  716. Nov 27 01:20:12 vs001 cib: [1012]: WARN: cib_server_process_diff: Not requesting full refresh in R/W mode
  717. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/189, version=0.629.362): ok (rc=0)
  718. Nov 27 01:20:12 vs001 crmd: [1016]: info: crmd_ais_dispatch: Setting expected votes to 3
  719. Nov 27 01:20:12 vs001 crmd: [1016]: WARN: crmd_ha_msg_filter: Another DC detected: vs003 (op=noop)
  720. Nov 27 01:20:12 vs001 crmd: [1016]: info: do_state_transition: State transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
  721. Nov 27 01:20:12 vs001 crmd: [1016]: info: update_dc: Unset DC vs001
  722. Nov 27 01:20:12 vs001 crmd: [1016]: info: do_state_transition: State transition S_ELECTION -> S_RELEASE_DC [ input=I_RELEASE_DC cause=C_FSA_INTERNAL origin=do_election_count_vote ]
  723. Nov 27 01:20:12 vs001 crmd: [1016]: info: do_dc_release: DC role released
  724. Nov 27 01:20:12 vs001 crmd: [1016]: info: pe_connection_destroy: Connection to the Policy Engine released
  725. Nov 27 01:20:12 vs001 crmd: [1016]: info: do_te_control: Transitioner is now inactive
  726. Nov 27 01:20:12 vs001 crmd: [1016]: info: do_state_transition: State transition S_RELEASE_DC -> S_PENDING [ input=I_RELEASE_SUCCESS cause=C_FSA_INTERNAL origin=do_dc_release ]
  727. Nov 27 01:20:12 vs001 crmd: [1016]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vs002']/lrm": ok (rc=0)
  728. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/191, version=0.629.363): ok (rc=0)
  729. Nov 27 01:20:12 vs001 crmd: [1016]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vs002']/transient_attributes": ok (rc=0)
  730. Nov 27 01:20:12 vs001 crmd: [1016]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vs003']/lrm": ok (rc=0)
  731. Nov 27 01:20:12 vs001 crmd: [1016]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vs003']/transient_attributes": ok (rc=0)
  732. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_readwrite: We are now in R/O mode
  733. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_process_diff: Diff 0.629.481 -> 0.629.482 not applied to 0.629.363: current "num_updates" is less than required
  734. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_server_process_diff: Requesting re-sync from peer
  735. Nov 27 01:20:12 vs001 cib: [1012]: notice: cib_server_process_diff: Not applying diff 0.629.482 -> 0.629.483 (sync in progress)
  736. Nov 27 01:20:12 vs001 cib: [1012]: notice: cib_server_process_diff: Not applying diff 0.629.483 -> 0.629.484 (sync in progress)
  737. Nov 27 01:20:12 vs001 crmd: [1016]: info: update_dc: Set DC to vs003 (3.0.5)
  738. Nov 27 01:20:12 vs001 cib: [1012]: info: cib_replace_notify: Replaced: -1.-1.-1 -> 0.629.484 from vs003
  739. Nov 27 01:20:12 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_ronon:1 (10000)
  740. Nov 27 01:20:12 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup:0 (10000)
  741. Nov 27 01:20:13 vs001 cib: [1012]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=vs003/crmd/7530, version=0.629.489): ok (rc=0)
  742. Nov 27 01:20:13 vs001 crmd: [1016]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
  743. Nov 27 01:20:13 vs001 kernel: [44244.835033] block drbd1: Handshake successful: Agreed network protocol version 96
  744. Nov 27 01:20:13 vs001 kernel: [44244.835040] block drbd1: conn( WFConnection -> WFReportParams )
  745. Nov 27 01:20:13 vs001 kernel: [44244.835055] block drbd1: Starting asender thread (from drbd1_receiver [4445])
  746. Nov 27 01:20:13 vs001 kernel: [44244.835141] block drbd1: data-integrity-alg: <not-used>
  747. Nov 27 01:20:13 vs001 kernel: [44244.835150] block drbd1: drbd_sync_handshake:
  748. Nov 27 01:20:13 vs001 kernel: [44244.835152] block drbd1: self 5B155798912405BD:799281900832AAC6:AFD08B5063E56D3E:AFCF8B5063E56D3F bits:0 flags:0
  749. Nov 27 01:20:13 vs001 kernel: [44244.835154] block drbd1: peer DEBF3E9EB2590515:799281900832AAC7:AFD08B5063E56D3F:AFCF8B5063E56D3F bits:244 flags:0
  750. Nov 27 01:20:13 vs001 kernel: [44244.835156] block drbd1: uuid_compare()=100 by rule 90
  751. Nov 27 01:20:13 vs001 kernel: [44244.835158] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1
  752. Nov 27 01:20:13 vs001 kernel: [44244.837490] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0)
  753. Nov 27 01:20:13 vs001 kernel: [44244.837499] block drbd1: Split-Brain detected but unresolved, dropping connection!
  754. Nov 27 01:20:13 vs001 kernel: [44244.837943] block drbd1: helper command: /sbin/drbdadm split-brain minor-1
  755. Nov 27 01:20:13 vs001 kernel: [44244.840031] block drbd1: helper command: /sbin/drbdadm split-brain minor-1 exit code 0 (0x0)
  756. Nov 27 01:20:13 vs001 kernel: [44244.840037] block drbd1: conn( WFReportParams -> Disconnecting )
  757. Nov 27 01:20:13 vs001 kernel: [44244.840041] block drbd1: error receiving ReportState, l: 4!
  758. Nov 27 01:20:13 vs001 kernel: [44244.840484] block drbd1: asender terminated
  759. Nov 27 01:20:13 vs001 kernel: [44244.840486] block drbd1: Terminating drbd1_asender
  760. Nov 27 01:20:13 vs001 kernel: [44244.840522] block drbd1: Connection closed
  761. Nov 27 01:20:13 vs001 kernel: [44244.840526] block drbd1: conn( Disconnecting -> StandAlone )
  762. Nov 27 01:20:13 vs001 kernel: [44244.840534] block drbd1: receiver terminated
  763. Nov 27 01:20:13 vs001 kernel: [44244.840535] block drbd1: Terminating drbd1_receiver
  764. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-vg_cinder-HDD1 (INFINITY)
  765. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc-data:1 (10000)
  766. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard-data:0 (10000)
  767. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_oneill:1 (10000)
  768. Nov 27 01:20:13 vs001 kernel: [44244.934046] block drbd2: Handshake successful: Agreed network protocol version 96
  769. Nov 27 01:20:13 vs001 kernel: [44244.934054] block drbd2: conn( WFConnection -> WFReportParams )
  770. Nov 27 01:20:13 vs001 kernel: [44244.934118] block drbd2: Starting asender thread (from drbd2_receiver [3447])
  771. Nov 27 01:20:13 vs001 kernel: [44244.934465] block drbd2: data-integrity-alg: <not-used>
  772. Nov 27 01:20:13 vs001 kernel: [44244.934477] block drbd2: drbd_sync_handshake:
  773. Nov 27 01:20:13 vs001 kernel: [44244.934480] block drbd2: self B6D48CC19119A6B7:0CEDB5E1042B389C:AF20123A68A2C4AC:AF1F123A68A2C4AD bits:0 flags:0
  774. Nov 27 01:20:13 vs001 kernel: [44244.934482] block drbd2: peer 4964FAF197AC385F:0CEDB5E1042B389D:AF20123A68A2C4AD:AF1F123A68A2C4AD bits:0 flags:0
  775. Nov 27 01:20:13 vs001 kernel: [44244.934485] block drbd2: uuid_compare()=100 by rule 90
  776. Nov 27 01:20:13 vs001 kernel: [44244.934488] block drbd2: helper command: /sbin/drbdadm initial-split-brain minor-2
  777. Nov 27 01:20:13 vs001 kernel: [44244.937607] block drbd2: helper command: /sbin/drbdadm initial-split-brain minor-2 exit code 0 (0x0)
  778. Nov 27 01:20:13 vs001 kernel: [44244.937611] block drbd2: Split-Brain detected but unresolved, dropping connection!
  779. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter:1 (10000)
  780. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
  781. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup-data:1 (10000)
  782. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter-data:1 (10000)
  783. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vg_cinder-HDD1 (1385477804)
  784. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc:1 (10000)
  785. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_cinder-HDD1:1 (10000)
  786. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_hammond:1 (10000)
  787. Nov 27 01:20:13 vs001 kernel: [44244.938083] block drbd2: helper command: /sbin/drbdadm split-brain minor-2
  788. Nov 27 01:20:13 vs001 kernel: [44244.940601] block drbd2: helper command: /sbin/drbdadm split-brain minor-2 exit code 0 (0x0)
  789. Nov 27 01:20:13 vs001 kernel: [44244.940608] block drbd2: conn( WFReportParams -> Disconnecting )
  790. Nov 27 01:20:13 vs001 kernel: [44244.940613] block drbd2: error receiving ReportState, l: 4!
  791. Nov 27 01:20:13 vs001 kernel: [44244.941061] block drbd2: asender terminated
  792. Nov 27 01:20:13 vs001 kernel: [44244.941063] block drbd2: Terminating drbd2_asender
  793. Nov 27 01:20:13 vs001 kernel: [44244.941104] block drbd2: Connection closed
  794. Nov 27 01:20:13 vs001 kernel: [44244.941109] block drbd2: conn( Disconnecting -> StandAlone )
  795. Nov 27 01:20:13 vs001 kernel: [44244.941119] block drbd2: receiver terminated
  796. Nov 27 01:20:13 vs001 kernel: [44244.941121] block drbd2: Terminating drbd2_receiver
  797. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_rush:1 (10000)
  798. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard:0 (10000)
  799. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
  800. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_ronon:1 (10000)
  801. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup:0 (10000)
  802. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-vg_cinder-HDD1 (INFINITY)
  803. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc-data:1 (10000)
  804. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard-data:0 (10000)
  805. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_oneill:1 (10000)
  806. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter:1 (10000)
  807. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
  808. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup-data:1 (10000)
  809. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter-data:1 (10000)
  810. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vg_cinder-HDD1 (1385477804)
  811. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc:1 (10000)
  812. Nov 27 01:20:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_cinder-HDD1:1 (10000)
  813. Nov 27 01:20:13 vs001 kernel: [44245.179030] block drbd16: Handshake successful: Agreed network protocol version 96
  814. Nov 27 01:20:13 vs001 kernel: [44245.179038] block drbd16: conn( WFConnection -> WFReportParams )
  815. Nov 27 01:20:13 vs001 kernel: [44245.179057] block drbd16: Starting asender thread (from drbd16_receiver [10417])
  816. Nov 27 01:20:13 vs001 kernel: [44245.179143] block drbd16: data-integrity-alg: <not-used>
  817. Nov 27 01:20:13 vs001 kernel: [44245.179161] block drbd16: drbd_sync_handshake:
  818. Nov 27 01:20:13 vs001 kernel: [44245.179164] block drbd16: self FDC1951AA7D37BFB:A1978A43833EBAD4:0001000000000004:0000000000000004 bits:0 flags:0
  819. Nov 27 01:20:13 vs001 kernel: [44245.179167] block drbd16: peer 6F2969E2AA1C55F5:A1978A43833EBAD5:0001000000000004:0000000000000004 bits:0 flags:0
  820. Nov 27 01:20:13 vs001 kernel: [44245.179169] block drbd16: uuid_compare()=100 by rule 90
  821. Nov 27 01:20:13 vs001 kernel: [44245.179172] block drbd16: helper command: /sbin/drbdadm initial-split-brain minor-16
  822. Nov 27 01:20:13 vs001 kernel: [44245.181568] block drbd16: helper command: /sbin/drbdadm initial-split-brain minor-16 exit code 0 (0x0)
  823. Nov 27 01:20:13 vs001 kernel: [44245.181572] block drbd16: Split-Brain detected but unresolved, dropping connection!
  824. Nov 27 01:20:13 vs001 kernel: [44245.182141] block drbd16: helper command: /sbin/drbdadm split-brain minor-16
  825. Nov 27 01:20:13 vs001 kernel: [44245.186849] block drbd16: helper command: /sbin/drbdadm split-brain minor-16 exit code 0 (0x0)
  826. Nov 27 01:20:13 vs001 kernel: [44245.186869] block drbd16: conn( WFReportParams -> Disconnecting )
  827. Nov 27 01:20:13 vs001 kernel: [44245.186874] block drbd16: error receiving ReportState, l: 4!
  828. Nov 27 01:20:13 vs001 kernel: [44245.187511] block drbd16: meta connection shut down by peer.
  829. Nov 27 01:20:13 vs001 kernel: [44245.188097] block drbd16: asender terminated
  830. Nov 27 01:20:13 vs001 kernel: [44245.188100] block drbd16: Terminating drbd16_asender
  831. Nov 27 01:20:13 vs001 kernel: [44245.188140] block drbd16: Connection closed
  832. Nov 27 01:20:13 vs001 kernel: [44245.188146] block drbd16: conn( Disconnecting -> StandAlone )
  833. Nov 27 01:20:13 vs001 kernel: [44245.188157] block drbd16: receiver terminated
  834. Nov 27 01:20:13 vs001 kernel: [44245.188158] block drbd16: Terminating drbd16_receiver
  835. Nov 27 01:20:14 vs001 kernel: [44245.845557] block drbd3: Handshake successful: Agreed network protocol version 96
  836. Nov 27 01:20:14 vs001 kernel: [44245.845566] block drbd3: conn( WFConnection -> WFReportParams )
  837. Nov 27 01:20:14 vs001 kernel: [44245.845588] block drbd3: Starting asender thread (from drbd3_receiver [3540])
  838. Nov 27 01:20:14 vs001 kernel: [44245.846027] block drbd3: data-integrity-alg: <not-used>
  839. Nov 27 01:20:14 vs001 kernel: [44245.846200] block drbd3: drbd_sync_handshake:
  840. Nov 27 01:20:14 vs001 kernel: [44245.846203] block drbd3: self 20548E8051DBC995:C35416C6C7C7F778:6D3197FD0EA3C69A:6D3097FD0EA3C69B bits:0 flags:0
  841. Nov 27 01:20:14 vs001 kernel: [44245.846206] block drbd3: peer AF4DE38BC31AC82F:C35416C6C7C7F779:6D3197FD0EA3C69B:6D3097FD0EA3C69B bits:28 flags:0
  842. Nov 27 01:20:14 vs001 kernel: [44245.846208] block drbd3: uuid_compare()=100 by rule 90
  843. Nov 27 01:20:14 vs001 kernel: [44245.846218] block drbd3: helper command: /sbin/drbdadm initial-split-brain minor-3
  844. Nov 27 01:20:14 vs001 kernel: [44245.848023] block drbd3: helper command: /sbin/drbdadm initial-split-brain minor-3 exit code 0 (0x0)
  845. Nov 27 01:20:14 vs001 kernel: [44245.848026] block drbd3: Split-Brain detected but unresolved, dropping connection!
  846. Nov 27 01:20:14 vs001 kernel: [44245.848649] block drbd3: helper command: /sbin/drbdadm split-brain minor-3
  847. Nov 27 01:20:14 vs001 kernel: [44245.850460] block drbd3: helper command: /sbin/drbdadm split-brain minor-3 exit code 0 (0x0)
  848. Nov 27 01:20:14 vs001 kernel: [44245.850465] block drbd3: conn( WFReportParams -> Disconnecting )
  849. Nov 27 01:20:14 vs001 kernel: [44245.850470] block drbd3: error receiving ReportState, l: 4!
  850. Nov 27 01:20:14 vs001 kernel: [44245.851064] block drbd3: asender terminated
  851. Nov 27 01:20:14 vs001 kernel: [44245.851068] block drbd3: Terminating drbd3_asender
  852. Nov 27 01:20:14 vs001 kernel: [44245.851152] block drbd3: Connection closed
  853. Nov 27 01:20:14 vs001 kernel: [44245.851157] block drbd3: conn( Disconnecting -> StandAlone )
  854. Nov 27 01:20:14 vs001 kernel: [44245.851196] block drbd3: receiver terminated
  855. Nov 27 01:20:14 vs001 kernel: [44245.851197] block drbd3: Terminating drbd3_receiver
  856. Nov 27 01:20:14 vs001 kernel: [44246.001460] block drbd27: Handshake successful: Agreed network protocol version 96
  857. Nov 27 01:20:14 vs001 kernel: [44246.001468] block drbd27: conn( WFConnection -> WFReportParams )
  858. Nov 27 01:20:14 vs001 kernel: [44246.001523] block drbd27: Starting asender thread (from drbd27_receiver [4077])
  859. Nov 27 01:20:14 vs001 kernel: [44246.001829] block drbd27: data-integrity-alg: <not-used>
  860. Nov 27 01:20:14 vs001 kernel: [44246.001841] block drbd27: drbd_sync_handshake:
  861. Nov 27 01:20:14 vs001 kernel: [44246.001844] block drbd27: self 704C531570446E6F:3B73B12444735996:A2619AFEB006CDA8:A2609AFEB006CDA9 bits:0 flags:0
  862. Nov 27 01:20:14 vs001 kernel: [44246.001846] block drbd27: peer 078375194AD7861F:3B73B12444735997:A2619AFEB006CDA9:A2609AFEB006CDA9 bits:34 flags:0
  863. Nov 27 01:20:14 vs001 kernel: [44246.001849] block drbd27: uuid_compare()=100 by rule 90
  864. Nov 27 01:20:14 vs001 kernel: [44246.001853] block drbd27: helper command: /sbin/drbdadm initial-split-brain minor-27
  865. Nov 27 01:20:14 vs001 kernel: [44246.003702] block drbd27: helper command: /sbin/drbdadm initial-split-brain minor-27 exit code 0 (0x0)
  866. Nov 27 01:20:14 vs001 kernel: [44246.003706] block drbd27: Split-Brain detected but unresolved, dropping connection!
  867. Nov 27 01:20:14 vs001 kernel: [44246.004277] block drbd27: helper command: /sbin/drbdadm split-brain minor-27
  868. Nov 27 01:20:14 vs001 kernel: [44246.006086] block drbd27: helper command: /sbin/drbdadm split-brain minor-27 exit code 0 (0x0)
  869. Nov 27 01:20:14 vs001 kernel: [44246.006092] block drbd27: conn( WFReportParams -> Disconnecting )
  870. Nov 27 01:20:14 vs001 kernel: [44246.006096] block drbd27: error receiving ReportState, l: 4!
  871. Nov 27 01:20:14 vs001 kernel: [44246.006746] block drbd27: meta connection shut down by peer.
  872. Nov 27 01:20:14 vs001 kernel: [44246.007062] block drbd31: Handshake successful: Agreed network protocol version 96
  873. Nov 27 01:20:14 vs001 kernel: [44246.007067] block drbd31: conn( WFConnection -> WFReportParams )
  874. Nov 27 01:20:14 vs001 kernel: [44246.007094] block drbd31: Starting asender thread (from drbd31_receiver [4707])
  875. Nov 27 01:20:14 vs001 kernel: [44246.007192] block drbd31: data-integrity-alg: <not-used>
  876. Nov 27 01:20:14 vs001 kernel: [44246.007201] block drbd31: drbd_sync_handshake:
  877. Nov 27 01:20:14 vs001 kernel: [44246.007203] block drbd31: self F970ACEF0F86767D:AEA81E5CB7BF6942:4BBB731F08D87EBA:4BBA731F08D87EBB bits:0 flags:0
  878. Nov 27 01:20:14 vs001 kernel: [44246.007206] block drbd31: peer 20C807C64EE4E0DB:AEA81E5CB7BF6943:4BBB731F08D87EBB:4BBA731F08D87EBB bits:98 flags:0
  879. Nov 27 01:20:14 vs001 kernel: [44246.007208] block drbd31: uuid_compare()=100 by rule 90
  880. Nov 27 01:20:14 vs001 kernel: [44246.007210] block drbd31: helper command: /sbin/drbdadm initial-split-brain minor-31
  881. Nov 27 01:20:14 vs001 kernel: [44246.007243] block drbd27: asender terminated
  882. Nov 27 01:20:14 vs001 kernel: [44246.007245] block drbd27: Terminating drbd27_asender
  883. Nov 27 01:20:14 vs001 kernel: [44246.007322] block drbd27: Connection closed
  884. Nov 27 01:20:14 vs001 kernel: [44246.007326] block drbd27: conn( Disconnecting -> StandAlone )
  885. Nov 27 01:20:14 vs001 kernel: [44246.007378] block drbd27: receiver terminated
  886. Nov 27 01:20:14 vs001 kernel: [44246.007380] block drbd27: Terminating drbd27_receiver
  887. Nov 27 01:20:14 vs001 kernel: [44246.009050] block drbd31: helper command: /sbin/drbdadm initial-split-brain minor-31 exit code 0 (0x0)
  888. Nov 27 01:20:14 vs001 kernel: [44246.009054] block drbd31: Split-Brain detected but unresolved, dropping connection!
  889. Nov 27 01:20:14 vs001 kernel: [44246.009696] block drbd31: helper command: /sbin/drbdadm split-brain minor-31
  890. Nov 27 01:20:14 vs001 kernel: [44246.011913] block drbd31: helper command: /sbin/drbdadm split-brain minor-31 exit code 0 (0x0)
  891. Nov 27 01:20:14 vs001 kernel: [44246.011919] block drbd31: conn( WFReportParams -> Disconnecting )
  892. Nov 27 01:20:14 vs001 kernel: [44246.011924] block drbd31: error receiving ReportState, l: 4!
  893. Nov 27 01:20:14 vs001 kernel: [44246.012540] block drbd31: asender terminated
  894. Nov 27 01:20:14 vs001 kernel: [44246.012542] block drbd31: Terminating drbd31_asender
  895. Nov 27 01:20:14 vs001 kernel: [44246.012631] block drbd31: Connection closed
  896. Nov 27 01:20:14 vs001 kernel: [44246.012636] block drbd31: conn( Disconnecting -> StandAlone )
  897. Nov 27 01:20:14 vs001 kernel: [44246.012674] block drbd31: receiver terminated
  898. Nov 27 01:20:14 vs001 kernel: [44246.012676] block drbd31: Terminating drbd31_receiver
  899. Nov 27 01:20:14 vs001 kernel: [44246.191091] block drbd0: Handshake successful: Agreed network protocol version 96
  900. Nov 27 01:20:14 vs001 kernel: [44246.191099] block drbd0: conn( WFConnection -> WFReportParams )
  901. Nov 27 01:20:14 vs001 kernel: [44246.191151] block drbd0: Starting asender thread (from drbd0_receiver [3384])
  902. Nov 27 01:20:14 vs001 kernel: [44246.195032] block drbd0: data-integrity-alg: <not-used>
  903. Nov 27 01:20:14 vs001 kernel: [44246.195058] block drbd0: drbd_sync_handshake:
  904. Nov 27 01:20:14 vs001 kernel: [44246.195061] block drbd0: self F00AE33D316E99AF:0C721E70E838AA92:B0BDE6667D9CF606:B0BCE6667D9CF607 bits:0 flags:0
  905. Nov 27 01:20:14 vs001 kernel: [44246.195064] block drbd0: peer 876DCA4FD1CD75C7:0C721E70E838AA93:B0BDE6667D9CF607:B0BCE6667D9CF607 bits:1487 flags:0
  906. Nov 27 01:20:14 vs001 kernel: [44246.195067] block drbd0: uuid_compare()=100 by rule 90
  907. Nov 27 01:20:14 vs001 kernel: [44246.195071] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
  908. Nov 27 01:20:14 vs001 kernel: [44246.197010] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
  909. Nov 27 01:20:14 vs001 kernel: [44246.197014] block drbd0: Split-Brain detected but unresolved, dropping connection!
  910. Nov 27 01:20:14 vs001 kernel: [44246.197696] block drbd0: helper command: /sbin/drbdadm split-brain minor-0
  911. Nov 27 01:20:14 vs001 kernel: [44246.200495] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
  912. Nov 27 01:20:14 vs001 kernel: [44246.200502] block drbd0: conn( WFReportParams -> Disconnecting )
  913. Nov 27 01:20:14 vs001 kernel: [44246.200507] block drbd0: error receiving ReportState, l: 4!
  914. Nov 27 01:20:14 vs001 kernel: [44246.201228] block drbd0: asender terminated
  915. Nov 27 01:20:14 vs001 kernel: [44246.201232] block drbd0: Terminating drbd0_asender
  916. Nov 27 01:20:14 vs001 kernel: [44246.201268] block drbd0: Connection closed
  917. Nov 27 01:20:14 vs001 kernel: [44246.201273] block drbd0: conn( Disconnecting -> StandAlone )
  918. Nov 27 01:20:14 vs001 kernel: [44246.201282] block drbd0: receiver terminated
  919. Nov 27 01:20:14 vs001 kernel: [44246.201284] block drbd0: Terminating drbd0_receiver
  920. Nov 27 01:20:14 vs001 kernel: [44246.251076] block drbd32: Handshake successful: Agreed network protocol version 96
  921. Nov 27 01:20:14 vs001 kernel: [44246.251083] block drbd32: conn( WFConnection -> WFReportParams )
  922. Nov 27 01:20:14 vs001 kernel: [44246.251190] block drbd32: Starting asender thread (from drbd32_receiver [4772])
  923. Nov 27 01:20:14 vs001 kernel: [44246.251290] block drbd32: data-integrity-alg: <not-used>
  924. Nov 27 01:20:14 vs001 kernel: [44246.251301] block drbd32: drbd_sync_handshake:
  925. Nov 27 01:20:14 vs001 kernel: [44246.251303] block drbd32: self 8AFAF16B682F0BC9:2ACC972D85254D92:0B73122907A5B5F0:0B72122907A5B5F1 bits:0 flags:0
  926. Nov 27 01:20:14 vs001 kernel: [44246.251306] block drbd32: peer 200578047EA4E695:2ACC972D85254D93:0B73122907A5B5F1:0B72122907A5B5F1 bits:13609 flags:0
  927. Nov 27 01:20:14 vs001 kernel: [44246.251308] block drbd32: uuid_compare()=100 by rule 90
  928. Nov 27 01:20:14 vs001 kernel: [44246.251311] block drbd32: helper command: /sbin/drbdadm initial-split-brain minor-32
  929. Nov 27 01:20:14 vs001 kernel: [44246.253184] block drbd32: helper command: /sbin/drbdadm initial-split-brain minor-32 exit code 0 (0x0)
  930. Nov 27 01:20:14 vs001 kernel: [44246.253187] block drbd32: Split-Brain detected but unresolved, dropping connection!
  931. Nov 27 01:20:14 vs001 kernel: [44246.253903] block drbd32: helper command: /sbin/drbdadm split-brain minor-32
  932. Nov 27 01:20:14 vs001 kernel: [44246.256043] block drbd32: helper command: /sbin/drbdadm split-brain minor-32 exit code 0 (0x0)
  933. Nov 27 01:20:14 vs001 kernel: [44246.256049] block drbd32: conn( WFReportParams -> Disconnecting )
  934. Nov 27 01:20:14 vs001 kernel: [44246.256053] block drbd32: error receiving ReportState, l: 4!
  935. Nov 27 01:20:14 vs001 kernel: [44246.256831] block drbd32: asender terminated
  936. Nov 27 01:20:14 vs001 kernel: [44246.256834] block drbd32: Terminating drbd32_asender
  937. Nov 27 01:20:14 vs001 kernel: [44246.256917] block drbd32: Connection closed
  938. Nov 27 01:20:14 vs001 kernel: [44246.256922] block drbd32: conn( Disconnecting -> StandAlone )
  939. Nov 27 01:20:14 vs001 kernel: [44246.256960] block drbd32: receiver terminated
  940. Nov 27 01:20:14 vs001 kernel: [44246.256962] block drbd32: Terminating drbd32_receiver
  941. Nov 27 01:20:14 vs001 cib: [1012]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 1240 ms (> 100 ms) before being called (GSource: 0x1908600)
  942. Nov 27 01:20:14 vs001 cib: [1012]: info: G_SIG_dispatch: started at 1722489353 should have started at 1722489229
  943. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_hammond:1 (10000)
  944. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_rush:1 (10000)
  945. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard:0 (10000)
  946. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  947. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  948. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vg_cinder-HDD1 (1385511613)
  949. Nov 27 01:20:14 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 587: last-failure-vg_cinder-HDD1=1385511613
  950. Nov 27 01:20:14 vs001 kernel: [44246.666418] block drbd21: Handshake successful: Agreed network protocol version 96
  951. Nov 27 01:20:14 vs001 kernel: [44246.666428] block drbd21: conn( WFConnection -> WFReportParams )
  952. Nov 27 01:20:14 vs001 kernel: [44246.666533] block drbd21: Starting asender thread (from drbd21_receiver [3966])
  953. Nov 27 01:20:14 vs001 kernel: [44246.666722] block drbd21: data-integrity-alg: <not-used>
  954. Nov 27 01:20:14 vs001 kernel: [44246.666736] block drbd21: drbd_sync_handshake:
  955. Nov 27 01:20:14 vs001 kernel: [44246.666740] block drbd21: self 0229ABADF49E173B:62FC58A2B4B5A44C:94E62243BFDE1B1E:94E52243BFDE1B1F bits:0 flags:0
  956. Nov 27 01:20:14 vs001 kernel: [44246.666744] block drbd21: peer 3855637A1BA683BF:62FC58A2B4B5A44D:94E62243BFDE1B1F:94E52243BFDE1B1F bits:25 flags:0
  957. Nov 27 01:20:14 vs001 kernel: [44246.666748] block drbd21: uuid_compare()=100 by rule 90
  958. Nov 27 01:20:14 vs001 kernel: [44246.666752] block drbd21: helper command: /sbin/drbdadm initial-split-brain minor-21
  959. Nov 27 01:20:14 vs001 kernel: [44246.670052] block drbd21: helper command: /sbin/drbdadm initial-split-brain minor-21 exit code 0 (0x0)
  960. Nov 27 01:20:14 vs001 kernel: [44246.670057] block drbd21: Split-Brain detected but unresolved, dropping connection!
  961. Nov 27 01:20:14 vs001 kernel: [44246.671140] block drbd21: helper command: /sbin/drbdadm split-brain minor-21
  962. Nov 27 01:20:14 vs001 kernel: [44246.674193] block drbd21: helper command: /sbin/drbdadm split-brain minor-21 exit code 0 (0x0)
  963. Nov 27 01:20:14 vs001 kernel: [44246.674201] block drbd21: conn( WFReportParams -> Disconnecting )
  964. Nov 27 01:20:14 vs001 kernel: [44246.674209] block drbd21: error receiving ReportState, l: 4!
  965. Nov 27 01:20:14 vs001 kernel: [44246.675340] block drbd21: asender terminated
  966. Nov 27 01:20:14 vs001 kernel: [44246.675346] block drbd21: Terminating drbd21_asender
  967. Nov 27 01:20:14 vs001 kernel: [44246.675531] block drbd21: Connection closed
  968. Nov 27 01:20:14 vs001 kernel: [44246.675540] block drbd21: conn( Disconnecting -> StandAlone )
  969. Nov 27 01:20:14 vs001 kernel: [44246.675707] block drbd21: receiver terminated
  970. Nov 27 01:20:14 vs001 kernel: [44246.675711] block drbd21: Terminating drbd21_receiver
  971. Nov 27 01:20:14 vs001 kernel: [44246.777070] block drbd30: Handshake successful: Agreed network protocol version 96
  972. Nov 27 01:20:14 vs001 kernel: [44246.777078] block drbd30: conn( WFConnection -> WFReportParams )
  973. Nov 27 01:20:14 vs001 kernel: [44246.777175] block drbd30: Starting asender thread (from drbd30_receiver [3965])
  974. Nov 27 01:20:14 vs001 kernel: [44246.777641] block drbd30: data-integrity-alg: <not-used>
  975. Nov 27 01:20:14 vs001 kernel: [44246.777867] block drbd30: drbd_sync_handshake:
  976. Nov 27 01:20:14 vs001 kernel: [44246.777873] block drbd30: self 43A9065AD827787B:BC2F5E64029D705A:148B2DB5AF7A3AF2:148A2DB5AF7A3AF3 bits:0 flags:0
  977. Nov 27 01:20:14 vs001 kernel: [44246.777877] block drbd30: peer D0CCDD2E10FD1549:BC2F5E64029D705B:148B2DB5AF7A3AF3:148A2DB5AF7A3AF3 bits:169 flags:0
  978. Nov 27 01:20:14 vs001 kernel: [44246.777881] block drbd30: uuid_compare()=100 by rule 90
  979. Nov 27 01:20:14 vs001 kernel: [44246.777885] block drbd30: helper command: /sbin/drbdadm initial-split-brain minor-30
  980. Nov 27 01:20:14 vs001 kernel: [44246.781130] block drbd30: helper command: /sbin/drbdadm initial-split-brain minor-30 exit code 0 (0x0)
  981. Nov 27 01:20:14 vs001 kernel: [44246.781135] block drbd30: Split-Brain detected but unresolved, dropping connection!
  982. Nov 27 01:20:14 vs001 kernel: [44246.782250] block drbd30: helper command: /sbin/drbdadm split-brain minor-30
  983. Nov 27 01:20:14 vs001 kernel: [44246.785522] block drbd30: helper command: /sbin/drbdadm split-brain minor-30 exit code 0 (0x0)
  984. Nov 27 01:20:14 vs001 kernel: [44246.785532] block drbd30: conn( WFReportParams -> Disconnecting )
  985. Nov 27 01:20:14 vs001 kernel: [44246.785540] block drbd30: error receiving ReportState, l: 4!
  986. Nov 27 01:20:14 vs001 kernel: [44246.786105] block drbd30: meta connection shut down by peer.
  987. Nov 27 01:20:14 vs001 kernel: [44246.786110] block drbd30: asender terminated
  988. Nov 27 01:20:14 vs001 kernel: [44246.786112] block drbd30: Terminating drbd30_asender
  989. Nov 27 01:20:14 vs001 kernel: [44246.787797] block drbd30: Connection closed
  990. Nov 27 01:20:14 vs001 kernel: [44246.787804] block drbd30: conn( Disconnecting -> StandAlone )
  991. Nov 27 01:20:14 vs001 kernel: [44246.787819] block drbd30: receiver terminated
  992. Nov 27 01:20:14 vs001 kernel: [44246.787825] block drbd30: Terminating drbd30_receiver
  993. Nov 27 01:20:15 vs001 kernel: [44247.562388] block drbd20: Handshake successful: Agreed network protocol version 96
  994. Nov 27 01:20:15 vs001 kernel: [44247.562404] block drbd20: conn( WFConnection -> WFReportParams )
  995. Nov 27 01:20:15 vs001 kernel: [44247.562434] block drbd20: Starting asender thread (from drbd20_receiver [3887])
  996. Nov 27 01:20:15 vs001 kernel: [44247.562616] block drbd20: data-integrity-alg: <not-used>
  997. Nov 27 01:20:15 vs001 kernel: [44247.562624] block drbd20: Considerable difference in lower level device sizes: 20970808s vs. 10485368s
  998. Nov 27 01:20:15 vs001 kernel: [44247.562635] block drbd20: drbd_sync_handshake:
  999. Nov 27 01:20:15 vs001 kernel: [44247.562639] block drbd20: self 7AAA1A5F1D99C87F:6A7320496FAF9214:E52B26590621112E:E52A26590621112F bits:0 flags:0
  1000. Nov 27 01:20:15 vs001 kernel: [44247.562643] block drbd20: peer 151A3BEB2BFA9C29:6A7320496FAF9215:E52B26590621112F:E52A26590621112F bits:92 flags:0
  1001. Nov 27 01:20:15 vs001 kernel: [44247.562646] block drbd20: uuid_compare()=100 by rule 90
  1002. Nov 27 01:20:15 vs001 kernel: [44247.562651] block drbd20: helper command: /sbin/drbdadm initial-split-brain minor-20
  1003. Nov 27 01:20:15 vs001 kernel: [44247.566271] block drbd20: helper command: /sbin/drbdadm initial-split-brain minor-20 exit code 0 (0x0)
  1004. Nov 27 01:20:15 vs001 kernel: [44247.566277] block drbd20: Split-Brain detected but unresolved, dropping connection!
  1005. Nov 27 01:20:15 vs001 kernel: [44247.567435] block drbd20: helper command: /sbin/drbdadm split-brain minor-20
  1006. Nov 27 01:20:15 vs001 kernel: [44247.570697] block drbd20: helper command: /sbin/drbdadm split-brain minor-20 exit code 0 (0x0)
  1007. Nov 27 01:20:15 vs001 kernel: [44247.570705] block drbd20: conn( WFReportParams -> Disconnecting )
  1008. Nov 27 01:20:15 vs001 kernel: [44247.570712] block drbd20: error receiving ReportState, l: 4!
  1009. Nov 27 01:20:15 vs001 kernel: [44247.571818] block drbd20: meta connection shut down by peer.
  1010. Nov 27 01:20:15 vs001 kernel: [44247.571823] block drbd20: asender terminated
  1011. Nov 27 01:20:15 vs001 kernel: [44247.571826] block drbd20: Terminating drbd20_asender
  1012. Nov 27 01:20:15 vs001 kernel: [44247.573069] block drbd20: Connection closed
  1013. Nov 27 01:20:15 vs001 kernel: [44247.573075] block drbd20: conn( Disconnecting -> StandAlone )
  1014. Nov 27 01:20:15 vs001 kernel: [44247.573095] block drbd20: receiver terminated
  1015. Nov 27 01:20:15 vs001 kernel: [44247.573098] block drbd20: Terminating drbd20_receiver
  1016. Nov 27 01:20:18 vs001 kernel: [44250.654917] block drbd4: Handshake successful: Agreed network protocol version 96
  1017. Nov 27 01:20:18 vs001 kernel: [44250.654927] block drbd4: conn( WFConnection -> WFReportParams )
  1018. Nov 27 01:20:18 vs001 kernel: [44250.654954] block drbd4: Starting asender thread (from drbd4_receiver [3570])
  1019. Nov 27 01:20:18 vs001 kernel: [44250.655109] block drbd4: data-integrity-alg: <not-used>
  1020. Nov 27 01:20:18 vs001 kernel: [44250.655123] block drbd4: drbd_sync_handshake:
  1021. Nov 27 01:20:18 vs001 kernel: [44250.655127] block drbd4: self 8C9AEB8D246A26CD:2205F7F942B19D7C:A6AA3C18C50817EA:A6A93C18C50817EB bits:0 flags:0
  1022. Nov 27 01:20:18 vs001 kernel: [44250.655162] block drbd4: peer A130864DB41B0A9F:2205F7F942B19D7D:A6AA3C18C50817EB:A6A93C18C50817EB bits:63 flags:0
  1023. Nov 27 01:20:18 vs001 kernel: [44250.655189] block drbd4: uuid_compare()=100 by rule 90
  1024. Nov 27 01:20:18 vs001 kernel: [44250.655199] block drbd4: helper command: /sbin/drbdadm initial-split-brain minor-4
  1025. Nov 27 01:20:18 vs001 kernel: [44250.658520] block drbd4: helper command: /sbin/drbdadm initial-split-brain minor-4 exit code 0 (0x0)
  1026. Nov 27 01:20:18 vs001 kernel: [44250.658526] block drbd4: Split-Brain detected but unresolved, dropping connection!
  1027. Nov 27 01:20:18 vs001 kernel: [44250.659815] block drbd4: helper command: /sbin/drbdadm split-brain minor-4
  1028. Nov 27 01:20:18 vs001 kernel: [44250.660353] block drbd4: meta connection shut down by peer.
  1029. Nov 27 01:20:18 vs001 kernel: [44250.661511] block drbd4: conn( WFReportParams -> NetworkFailure )
  1030. Nov 27 01:20:18 vs001 kernel: [44250.661519] block drbd4: asender terminated
  1031. Nov 27 01:20:18 vs001 kernel: [44250.661523] block drbd4: Terminating drbd4_asender
  1032. Nov 27 01:20:18 vs001 kernel: [44250.663032] block drbd4: helper command: /sbin/drbdadm split-brain minor-4 exit code 0 (0x0)
  1033. Nov 27 01:20:18 vs001 kernel: [44250.663041] block drbd4: conn( NetworkFailure -> Disconnecting )
  1034. Nov 27 01:20:18 vs001 kernel: [44250.663049] block drbd4: error receiving ReportState, l: 4!
  1035. Nov 27 01:20:18 vs001 kernel: [44250.664378] block drbd4: Connection closed
  1036. Nov 27 01:20:18 vs001 kernel: [44250.664387] block drbd4: conn( Disconnecting -> StandAlone )
  1037. Nov 27 01:20:18 vs001 kernel: [44250.664410] block drbd4: receiver terminated
  1038. Nov 27 01:20:18 vs001 kernel: [44250.664413] block drbd4: Terminating drbd4_receiver
  1039. Nov 27 01:20:21 vs001 kernel: [44253.506891] block drbd7: Handshake successful: Agreed network protocol version 96
  1040. Nov 27 01:20:21 vs001 kernel: [44253.506902] block drbd7: conn( WFConnection -> WFReportParams )
  1041. Nov 27 01:20:21 vs001 kernel: [44253.507008] block drbd7: Starting asender thread (from drbd7_receiver [4701])
  1042. Nov 27 01:20:21 vs001 kernel: [44253.507205] block drbd7: data-integrity-alg: <not-used>
  1043. Nov 27 01:20:21 vs001 kernel: [44253.507221] block drbd7: drbd_sync_handshake:
  1044. Nov 27 01:20:21 vs001 kernel: [44253.507225] block drbd7: self 5A996DCBBB8B8D65:9F4F613EF03DCD86:7C58AB812E3670D6:7C57AB812E3670D7 bits:0 flags:0
  1045. Nov 27 01:20:21 vs001 kernel: [44253.507229] block drbd7: peer EF5A53B2288D0401:9F4F613EF03DCD87:7C58AB812E3670D7:7C57AB812E3670D7 bits:9 flags:0
  1046. Nov 27 01:20:21 vs001 kernel: [44253.507232] block drbd7: uuid_compare()=100 by rule 90
  1047. Nov 27 01:20:21 vs001 kernel: [44253.507237] block drbd7: helper command: /sbin/drbdadm initial-split-brain minor-7
  1048. Nov 27 01:20:21 vs001 kernel: [44253.510655] block drbd7: helper command: /sbin/drbdadm initial-split-brain minor-7 exit code 0 (0x0)
  1049. Nov 27 01:20:21 vs001 kernel: [44253.510661] block drbd7: Split-Brain detected but unresolved, dropping connection!
  1050. Nov 27 01:20:21 vs001 kernel: [44253.511837] block drbd7: helper command: /sbin/drbdadm split-brain minor-7
  1051. Nov 27 01:20:21 vs001 kernel: [44253.512869] block drbd7: meta connection shut down by peer.
  1052. Nov 27 01:20:21 vs001 kernel: [44253.514139] block drbd7: conn( WFReportParams -> NetworkFailure )
  1053. Nov 27 01:20:21 vs001 kernel: [44253.514152] block drbd7: asender terminated
  1054. Nov 27 01:20:21 vs001 kernel: [44253.514155] block drbd7: Terminating drbd7_asender
  1055. Nov 27 01:20:21 vs001 kernel: [44253.515114] block drbd7: helper command: /sbin/drbdadm split-brain minor-7 exit code 0 (0x0)
  1056. Nov 27 01:20:21 vs001 kernel: [44253.515121] block drbd7: conn( NetworkFailure -> Disconnecting )
  1057. Nov 27 01:20:21 vs001 kernel: [44253.515128] block drbd7: error receiving ReportState, l: 4!
  1058. Nov 27 01:20:21 vs001 kernel: [44253.516359] block drbd7: Connection closed
  1059. Nov 27 01:20:21 vs001 kernel: [44253.516365] block drbd7: conn( Disconnecting -> StandAlone )
  1060. Nov 27 01:20:21 vs001 kernel: [44253.516381] block drbd7: receiver terminated
  1061. Nov 27 01:20:21 vs001 kernel: [44253.516384] block drbd7: Terminating drbd7_receiver
  1062. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[200] on drbd_rush:1 for client 1016, its parameters: drbd_resource=[rush] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_rush:1 ] CRM_meta_notify_inactive_resource=[drbd_rush:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone_node_max=[1 cancelled
  1063. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_rush:1_monitor_29000 from 8:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-194
  1064. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[198] on drbd_ronon:1 for client 1016, its parameters: drbd_resource=[ronon-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_ronon:1 ] CRM_meta_notify_inactive_resource=[drbd_ronon:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clone_no cancelled
  1065. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_monitor_29000 from 12:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-195
  1066. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[204] on drbd_hammond:1 for client 1016, its parameters: drbd_resource=[hammond-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_hammond:1 ] CRM_meta_notify_inactive_resource=[drbd_hammond:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_ cancelled
  1067. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_hammond:1_monitor_29000 from 1:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-196
  1068. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[203] on drbd_carter:1 for client 1016, its parameters: drbd_resource=[carter-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_carter:1 ] CRM_meta_notify_inactive_resource=[drbd_carter:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clon cancelled
  1069. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_monitor_29000 from 6:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-197
  1070. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[209] on drbd_carter-data:1 for client 1016, its parameters: drbd_resource=[carter-data] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_carter-data:1 ] CRM_meta_notify_inactive_resource=[drbd_carter-data:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 cancelled
  1071. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter-data:1_monitor_29000 from 5:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-198
  1072. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[210] on drbd_oneill:1 for client 1016, its parameters: drbd_resource=[oneill-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_oneill:1 ] CRM_meta_notify_inactive_resource=[drbd_oneill:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clon cancelled
  1073. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_oneill:1_monitor_29000 from 7:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-199
  1074. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[217] on drbd_backup:0 for client 1016, its parameters: drbd_resource=[backup-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_backup:0 ] CRM_meta_notify_inactive_resource=[drbd_backup:0 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_meta_clon cancelled
  1075. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_monitor_29000 from 2:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-200
  1076. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[224] on drbd_backup-data:1 for client 1016, its parameters: drbd_resource=[backup-data] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_backup-data:1 ] CRM_meta_notify_inactive_resource=[drbd_backup-data:1 ] CRM_meta_notify_master_uname=[vs003 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 cancelled
  1077. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_monitor_29000 from 13:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-201
  1078. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[215] on drbd_sheppard:0 for client 1016, its parameters: drbd_resource=[sheppard-root] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_sheppard:0 ] CRM_meta_notify_inactive_resource=[drbd_sheppard:0 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM_m cancelled
  1079. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_monitor_29000 from 11:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-202
  1080. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[220] on drbd_sheppard-data:0 for client 1016, its parameters: drbd_resource=[sheppard-data] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_sheppard-data:0 ] CRM_meta_notify_inactive_resource=[drbd_sheppard-data:0 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname cancelled
  1081. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_monitor_29000 from 3:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-203
  1082. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[227] on drbd_tealc:1 for client 1016, its parameters: drbd_resource=[tealc-root2] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_tealc:1 ] CRM_meta_notify_inactive_resource=[drbd_tealc:1 drbd_tealc:2 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[vs001 ] CRM cancelled
  1083. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_monitor_29000 from 4:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-204
  1084. Nov 27 01:21:47 vs001 lrmd: [1013]: info: cancel_op: operation monitor[231] on drbd_tealc-data:1 for client 1016, its parameters: drbd_resource=[tealc-data2] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_tealc-data:1 ] CRM_meta_notify_inactive_resource=[drbd_tealc-data:1 drbd_tealc-data:2 ] CRM_meta_notify_master_uname=[vs002 ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CRM_meta_notify_star cancelled
  1085. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_monitor_29000 from 9:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-205
  1086. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=684:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_rush:1_notify_0 )
  1087. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 notify[232] (pid 3009)
  1088. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=700:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_ronon:1_notify_0 )
  1089. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 notify[233] (pid 3010)
  1090. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=716:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_hammond:1_notify_0 )
  1091. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 notify[234] (pid 3011)
  1092. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=732:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter:1_notify_0 )
  1093. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 notify[235] (pid 3012)
  1094. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=748:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter-data:1_notify_0 )
  1095. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_monitor_29000 (call=200, status=1, cib-update=0, confirmed=true) Cancelled
  1096. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_monitor_29000 (call=198, status=1, cib-update=0, confirmed=true) Cancelled
  1097. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_monitor_29000 (call=204, status=1, cib-update=0, confirmed=true) Cancelled
  1098. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_monitor_29000 (call=203, status=1, cib-update=0, confirmed=true) Cancelled
  1099. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_monitor_29000 (call=209, status=1, cib-update=0, confirmed=true) Cancelled
  1100. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_monitor_29000 (call=210, status=1, cib-update=0, confirmed=true) Cancelled
  1101. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_monitor_29000 (call=217, status=1, cib-update=0, confirmed=true) Cancelled
  1102. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_monitor_29000 (call=224, status=1, cib-update=0, confirmed=true) Cancelled
  1103. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_monitor_29000 (call=215, status=1, cib-update=0, confirmed=true) Cancelled
  1104. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_monitor_29000 (call=220, status=1, cib-update=0, confirmed=true) Cancelled
  1105. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_monitor_29000 (call=227, status=1, cib-update=0, confirmed=true) Cancelled
  1106. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_monitor_29000 (call=231, status=1, cib-update=0, confirmed=true) Cancelled
  1107. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=764:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_oneill:1_notify_0 )
  1108. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=778:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_notify_0 )
  1109. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=796:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_notify_0 )
  1110. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=826:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1111. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=842:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_notify_0 )
  1112. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=860:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1113. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=876:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1114. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=908:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1115. Nov 27 01:21:47 vs001 lrmd: [1013]: info: operation notify[233] on drbd_ronon:1 for client 1016: pid 3010 exited with return code 0
  1116. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_notify_0 from 700:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-206
  1117. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_notify_0 (call=233, rc=0, cib-update=0, confirmed=true) ok
  1118. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=90:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_ronon:1_demote_0 )
  1119. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 demote[245] (pid 3100)
  1120. Nov 27 01:21:47 vs001 lrmd: [1013]: info: operation notify[234] on drbd_hammond:1 for client 1016: pid 3011 exited with return code 0
  1121. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_hammond:1_notify_0 from 716:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-207
  1122. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_notify_0 (call=234, rc=0, cib-update=0, confirmed=true) ok
  1123. Nov 27 01:21:47 vs001 lrmd: [1013]: info: operation notify[235] on drbd_carter:1 for client 1016: pid 3012 exited with return code 0
  1124. Nov 27 01:21:47 vs001 lrmd: [1013]: info: operation notify[232] on drbd_rush:1 for client 1016: pid 3009 exited with return code 0
  1125. Nov 27 01:21:47 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=130:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_hammond:1_demote_0 )
  1126. Nov 27 01:21:47 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 demote[246] (pid 3115)
  1127. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_notify_0 from 732:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-208
  1128. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_notify_0 (call=235, rc=0, cib-update=0, confirmed=true) ok
  1129. Nov 27 01:21:47 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_rush:1_notify_0 from 684:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511707-209
  1130. Nov 27 01:21:47 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_notify_0 (call=232, rc=0, cib-update=0, confirmed=true) ok
  1131. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=50:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_rush:1_demote_0 )
  1132. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 demote[247] (pid 3124)
  1133. Nov 27 01:21:48 vs001 kernel: [44339.805565] block drbd4: role( Primary -> Secondary )
  1134. Nov 27 01:21:48 vs001 kernel: [44339.805584] block drbd4: bitmap WRITE of 0 pages took 0 jiffies
  1135. Nov 27 01:21:48 vs001 kernel: [44339.816187] block drbd2: role( Primary -> Secondary )
  1136. Nov 27 01:21:48 vs001 kernel: [44339.816210] block drbd2: bitmap WRITE of 0 pages took 0 jiffies
  1137. Nov 27 01:21:48 vs001 kernel: [44339.822831] block drbd0: role( Primary -> Secondary )
  1138. Nov 27 01:21:48 vs001 kernel: [44339.822846] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  1139. Nov 27 01:21:48 vs001 kernel: [44339.845608] block drbd4: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1140. Nov 27 01:21:48 vs001 kernel: [44339.845721] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1141. Nov 27 01:21:48 vs001 kernel: [44339.845753] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1142. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_hammond:1:demote:stdout)
  1143. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_rush:1:demote:stdout)
  1144. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation demote[246] on drbd_hammond:1 for client 1016: pid 3115 exited with return code 0
  1145. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation demote[247] on drbd_rush:1 for client 1016: pid 3124 exited with return code 0
  1146. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation demote[245] on drbd_ronon:1 for client 1016: pid 3100 exited with return code 0
  1147. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_ronon:1:demote:stdout)
  1148. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_demote_0 (call=246, rc=0, cib-update=208, confirmed=true) ok
  1149. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_demote_0 (call=247, rc=0, cib-update=209, confirmed=true) ok
  1150. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_demote_0 (call=245, rc=0, cib-update=210, confirmed=true) ok
  1151. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=717:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_hammond:1_notify_0 )
  1152. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 notify[248] (pid 3245)
  1153. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=685:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_rush:1_notify_0 )
  1154. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 notify[249] (pid 3266)
  1155. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=701:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_ronon:1_notify_0 )
  1156. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 notify[250] (pid 3279)
  1157. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_rush:1 (1000)
  1158. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 591: master-drbd_rush:1=1000
  1159. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_hammond:1:notify:stdout)
  1160. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[248] on drbd_hammond:1 for client 1016: pid 3245 exited with return code 0
  1161. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_rush:1:notify:stdout)
  1162. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_hammond:1_notify_0 from 717:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-210
  1163. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_hammond:1 (1000)
  1164. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_notify_0 (call=248, rc=0, cib-update=0, confirmed=true) ok
  1165. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[249] on drbd_rush:1 for client 1016: pid 3266 exited with return code 0
  1166. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 593: master-drbd_hammond:1=1000
  1167. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_rush:1_notify_0 from 685:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-211
  1168. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_notify_0 (call=249, rc=0, cib-update=0, confirmed=true) ok
  1169. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_ronon:1 (1000)
  1170. Nov 27 01:21:48 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 595: master-drbd_ronon:1=1000
  1171. Nov 27 01:21:48 vs001 lrmd: [1013]: info: RA output: (drbd_ronon:1:notify:stdout)
  1172. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[250] on drbd_ronon:1 for client 1016: pid 3279 exited with return code 0
  1173. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=131:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_hammond:1_monitor_31000 )
  1174. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_hammond:1 monitor[251] (pid 3340)
  1175. Nov 27 01:21:48 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=51:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_rush:1_monitor_31000 )
  1176. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_rush:1 monitor[252] (pid 3341)
  1177. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_ronon:1_notify_0 from 701:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-212
  1178. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_notify_0 (call=250, rc=0, cib-update=0, confirmed=true) ok
  1179. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation monitor[251] on drbd_hammond:1 for client 1016: pid 3340 exited with return code 0
  1180. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation monitor[252] on drbd_rush:1 for client 1016: pid 3341 exited with return code 0
  1181. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_hammond:1_monitor_31000 (call=251, rc=0, cib-update=211, confirmed=false) ok
  1182. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_rush:1_monitor_31000 (call=252, rc=0, cib-update=212, confirmed=false) ok
  1183. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 notify[236] (pid 3399)
  1184. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 notify[237] (pid 3400)
  1185. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[238] (pid 3409)
  1186. Nov 27 01:21:48 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[239] (pid 3411)
  1187. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[237] on drbd_oneill:1 for client 1016: pid 3400 exited with return code 0
  1188. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[236] on drbd_carter-data:1 for client 1016: pid 3399 exited with return code 0
  1189. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_oneill:1_notify_0 from 764:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-213
  1190. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_notify_0 (call=237, rc=0, cib-update=0, confirmed=true) ok
  1191. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter-data:1_notify_0 from 748:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-214
  1192. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_notify_0 (call=236, rc=0, cib-update=0, confirmed=true) ok
  1193. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[238] on drbd_backup:0 for client 1016: pid 3409 exited with return code 0
  1194. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_notify_0 from 778:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-215
  1195. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_notify_0 (call=238, rc=0, cib-update=0, confirmed=true) ok
  1196. Nov 27 01:21:48 vs001 lrmd: [1013]: info: operation notify[239] on drbd_backup-data:1 for client 1016: pid 3411 exited with return code 0
  1197. Nov 27 01:21:48 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_notify_0 from 796:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511708-216
  1198. Nov 27 01:21:48 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_notify_0 (call=239, rc=0, cib-update=0, confirmed=true) ok
  1199. Nov 27 01:21:49 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[240] (pid 3520)
  1200. Nov 27 01:21:49 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[241] (pid 3521)
  1201. Nov 27 01:21:49 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[242] (pid 3522)
  1202. Nov 27 01:21:49 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[243] (pid 3523)
  1203. Nov 27 01:21:49 vs001 lrmd: [1013]: info: operation notify[241] on drbd_sheppard-data:0 for client 1016: pid 3521 exited with return code 0
  1204. Nov 27 01:21:49 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_notify_0 from 842:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511709-217
  1205. Nov 27 01:21:49 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_notify_0 (call=241, rc=0, cib-update=0, confirmed=true) ok
  1206. Nov 27 01:21:49 vs001 lrmd: [1013]: info: operation notify[243] on drbd_tealc-data:1 for client 1016: pid 3523 exited with return code 0
  1207. Nov 27 01:21:49 vs001 lrmd: [1013]: info: operation notify[240] on drbd_sheppard:0 for client 1016: pid 3520 exited with return code 0
  1208. Nov 27 01:21:49 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 876:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511709-218
  1209. Nov 27 01:21:49 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=243, rc=0, cib-update=0, confirmed=true) ok
  1210. Nov 27 01:21:49 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 826:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511709-219
  1211. Nov 27 01:21:49 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=240, rc=0, cib-update=0, confirmed=true) ok
  1212. Nov 27 01:21:49 vs001 lrmd: [1013]: info: operation notify[242] on drbd_tealc:1 for client 1016: pid 3522 exited with return code 0
  1213. Nov 27 01:21:49 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 860:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511709-220
  1214. Nov 27 01:21:49 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=242, rc=0, cib-update=0, confirmed=true) ok
  1215. Nov 27 01:21:50 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[244] (pid 3612)
  1216. Nov 27 01:21:50 vs001 lrmd: [1013]: info: operation notify[244] on drbd_cinder-HDD1:1 for client 1016: pid 3612 exited with return code 0
  1217. Nov 27 01:21:50 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 908:4855:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511710-221
  1218. Nov 27 01:21:50 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=244, rc=0, cib-update=0, confirmed=true) ok
  1219. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=78:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_ronon:1_monitor_31000 )
  1220. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_ronon:1 monitor[253] (pid 3635)
  1221. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=717:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter:1_notify_0 )
  1222. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 notify[254] (pid 3636)
  1223. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=733:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter-data:1_notify_0 )
  1224. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 notify[255] (pid 3637)
  1225. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=749:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_oneill:1_notify_0 )
  1226. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 notify[256] (pid 3643)
  1227. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=763:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_notify_0 )
  1228. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=781:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_notify_0 )
  1229. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=811:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1230. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=827:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_notify_0 )
  1231. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=845:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1232. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=861:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1233. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=893:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1234. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[254] on drbd_carter:1 for client 1016: pid 3636 exited with return code 0
  1235. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_notify_0 from 717:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-222
  1236. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_notify_0 (call=254, rc=0, cib-update=0, confirmed=true) ok
  1237. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[255] on drbd_carter-data:1 for client 1016: pid 3637 exited with return code 0
  1238. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter-data:1_notify_0 from 733:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-223
  1239. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_notify_0 (call=255, rc=0, cib-update=0, confirmed=true) ok
  1240. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[256] on drbd_oneill:1 for client 1016: pid 3643 exited with return code 0
  1241. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=188:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter-data:1_demote_0 )
  1242. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 demote[264] (pid 3731)
  1243. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_oneill:1_notify_0 from 749:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-224
  1244. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_notify_0 (call=256, rc=0, cib-update=0, confirmed=true) ok
  1245. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation monitor[253] on drbd_ronon:1 for client 1016: pid 3635 exited with return code 0
  1246. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_ronon:1_monitor_31000 (call=253, rc=0, cib-update=213, confirmed=false) ok
  1247. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=228:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_oneill:1_demote_0 )
  1248. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 demote[265] (pid 3749)
  1249. Nov 27 01:21:51 vs001 kernel: [44343.018955] block drbd7: role( Primary -> Secondary )
  1250. Nov 27 01:21:51 vs001 kernel: [44343.018986] block drbd7: bitmap WRITE of 0 pages took 0 jiffies
  1251. Nov 27 01:21:51 vs001 kernel: [44343.039601] block drbd1: role( Primary -> Secondary )
  1252. Nov 27 01:21:51 vs001 kernel: [44343.039617] block drbd1: bitmap WRITE of 0 pages took 0 jiffies
  1253. Nov 27 01:21:51 vs001 kernel: [44343.083639] block drbd7: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1254. Nov 27 01:21:51 vs001 kernel: [44343.083678] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1255. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_carter-data:1:demote:stdout)
  1256. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_oneill:1:demote:stdout)
  1257. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation demote[264] on drbd_carter-data:1 for client 1016: pid 3731 exited with return code 0
  1258. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_demote_0 (call=264, rc=0, cib-update=214, confirmed=true) ok
  1259. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation demote[265] on drbd_oneill:1 for client 1016: pid 3749 exited with return code 0
  1260. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_demote_0 (call=265, rc=0, cib-update=215, confirmed=true) ok
  1261. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=734:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter-data:1_notify_0 )
  1262. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 notify[266] (pid 3816)
  1263. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=750:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_oneill:1_notify_0 )
  1264. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 notify[267] (pid 3817)
  1265. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter-data:1 (1000)
  1266. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 597: master-drbd_carter-data:1=1000
  1267. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_carter-data:1:notify:stdout)
  1268. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[266] on drbd_carter-data:1 for client 1016: pid 3816 exited with return code 0
  1269. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_oneill:1 (1000)
  1270. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter-data:1_notify_0 from 734:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-225
  1271. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_notify_0 (call=266, rc=0, cib-update=0, confirmed=true) ok
  1272. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 599: master-drbd_oneill:1=1000
  1273. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_oneill:1:notify:stdout)
  1274. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[267] on drbd_oneill:1 for client 1016: pid 3817 exited with return code 0
  1275. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=189:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter-data:1_monitor_31000 )
  1276. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter-data:1 monitor[268] (pid 3874)
  1277. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=156:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter:1_demote_0 )
  1278. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 demote[269] (pid 3875)
  1279. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_oneill:1_notify_0 from 750:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-226
  1280. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_notify_0 (call=267, rc=0, cib-update=0, confirmed=true) ok
  1281. Nov 27 01:21:51 vs001 kernel: [44343.377561] block drbd3: role( Primary -> Secondary )
  1282. Nov 27 01:21:51 vs001 kernel: [44343.377580] block drbd3: bitmap WRITE of 0 pages took 0 jiffies
  1283. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation monitor[268] on drbd_carter-data:1 for client 1016: pid 3874 exited with return code 0
  1284. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter-data:1_monitor_31000 (call=268, rc=0, cib-update=216, confirmed=false) ok
  1285. Nov 27 01:21:51 vs001 kernel: [44343.403252] block drbd3: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1286. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_carter:1:demote:stdout)
  1287. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation demote[269] on drbd_carter:1 for client 1016: pid 3875 exited with return code 0
  1288. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_demote_0 (call=269, rc=0, cib-update=217, confirmed=true) ok
  1289. Nov 27 01:21:51 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=718:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter:1_notify_0 )
  1290. Nov 27 01:21:51 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 notify[270] (pid 3940)
  1291. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_carter:1 (1000)
  1292. Nov 27 01:21:51 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 601: master-drbd_carter:1=1000
  1293. Nov 27 01:21:51 vs001 lrmd: [1013]: info: RA output: (drbd_carter:1:notify:stdout)
  1294. Nov 27 01:21:51 vs001 lrmd: [1013]: info: operation notify[270] on drbd_carter:1 for client 1016: pid 3940 exited with return code 0
  1295. Nov 27 01:21:51 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_carter:1_notify_0 from 718:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511711-227
  1296. Nov 27 01:21:51 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_notify_0 (call=270, rc=0, cib-update=0, confirmed=true) ok
  1297. Nov 27 01:21:52 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[257] (pid 3968)
  1298. Nov 27 01:21:52 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[258] (pid 3969)
  1299. Nov 27 01:21:52 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[259] (pid 3970)
  1300. Nov 27 01:21:52 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[260] (pid 3971)
  1301. Nov 27 01:21:52 vs001 lrmd: [1013]: info: operation notify[260] on drbd_sheppard-data:0 for client 1016: pid 3971 exited with return code 0
  1302. Nov 27 01:21:52 vs001 lrmd: [1013]: info: operation notify[257] on drbd_backup:0 for client 1016: pid 3968 exited with return code 0
  1303. Nov 27 01:21:52 vs001 lrmd: [1013]: info: operation notify[258] on drbd_backup-data:1 for client 1016: pid 3969 exited with return code 0
  1304. Nov 27 01:21:52 vs001 lrmd: [1013]: info: operation notify[259] on drbd_sheppard:0 for client 1016: pid 3970 exited with return code 0
  1305. Nov 27 01:21:52 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_notify_0 from 827:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511712-228
  1306. Nov 27 01:21:52 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_notify_0 (call=260, rc=0, cib-update=0, confirmed=true) ok
  1307. Nov 27 01:21:52 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_notify_0 from 763:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511712-229
  1308. Nov 27 01:21:52 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_notify_0 (call=257, rc=0, cib-update=0, confirmed=true) ok
  1309. Nov 27 01:21:52 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_notify_0 from 781:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511712-230
  1310. Nov 27 01:21:52 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_notify_0 (call=258, rc=0, cib-update=0, confirmed=true) ok
  1311. Nov 27 01:21:52 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 811:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511712-231
  1312. Nov 27 01:21:52 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=259, rc=0, cib-update=0, confirmed=true) ok
  1313. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[261] (pid 4060)
  1314. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[262] (pid 4061)
  1315. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[263] (pid 4062)
  1316. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[262] on drbd_tealc-data:1 for client 1016: pid 4061 exited with return code 0
  1317. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[261] on drbd_tealc:1 for client 1016: pid 4060 exited with return code 0
  1318. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 861:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-232
  1319. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=262, rc=0, cib-update=0, confirmed=true) ok
  1320. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 845:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-233
  1321. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=261, rc=0, cib-update=0, confirmed=true) ok
  1322. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[263] on drbd_cinder-HDD1:1 for client 1016: pid 4062 exited with return code 0
  1323. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 893:4856:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-234
  1324. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=263, rc=0, cib-update=0, confirmed=true) ok
  1325. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=158:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_carter:1_monitor_31000 )
  1326. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_carter:1 monitor[271] (pid 4129)
  1327. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=229:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_oneill:1_monitor_31000 )
  1328. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_oneill:1 monitor[272] (pid 4130)
  1329. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=762:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_notify_0 )
  1330. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[273] (pid 4131)
  1331. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=780:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_notify_0 )
  1332. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[274] (pid 4132)
  1333. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=810:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1334. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=826:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_notify_0 )
  1335. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=844:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1336. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=860:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1337. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=892:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1338. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[273] on drbd_backup:0 for client 1016: pid 4131 exited with return code 0
  1339. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation monitor[271] on drbd_carter:1 for client 1016: pid 4129 exited with return code 0
  1340. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_notify_0 from 762:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-235
  1341. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_notify_0 (call=273, rc=0, cib-update=0, confirmed=true) ok
  1342. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_carter:1_monitor_31000 (call=271, rc=0, cib-update=218, confirmed=false) ok
  1343. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[274] on drbd_backup-data:1 for client 1016: pid 4132 exited with return code 0
  1344. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_notify_0 from 780:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-236
  1345. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_notify_0 (call=274, rc=0, cib-update=0, confirmed=true) ok
  1346. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=300:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_demote_0 )
  1347. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 demote[280] (pid 4232)
  1348. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation monitor[272] on drbd_oneill:1 for client 1016: pid 4130 exited with return code 0
  1349. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_oneill:1_monitor_31000 (call=272, rc=0, cib-update=219, confirmed=false) ok
  1350. Nov 27 01:21:53 vs001 kernel: [44345.202271] block drbd32: role( Primary -> Secondary )
  1351. Nov 27 01:21:53 vs001 kernel: [44345.202343] block drbd32: bitmap WRITE of 0 pages took 0 jiffies
  1352. Nov 27 01:21:53 vs001 kernel: [44345.273800] block drbd32: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1353. Nov 27 01:21:53 vs001 lrmd: [1013]: info: RA output: (drbd_backup-data:1:demote:stdout)
  1354. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation demote[280] on drbd_backup-data:1 for client 1016: pid 4232 exited with return code 0
  1355. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_demote_0 (call=280, rc=0, cib-update=220, confirmed=true) ok
  1356. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=781:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_notify_0 )
  1357. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 notify[281] (pid 4276)
  1358. Nov 27 01:21:53 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup-data:1 (1000)
  1359. Nov 27 01:21:53 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 603: master-drbd_backup-data:1=1000
  1360. Nov 27 01:21:53 vs001 lrmd: [1013]: info: RA output: (drbd_backup-data:1:notify:stdout)
  1361. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[281] on drbd_backup-data:1 for client 1016: pid 4276 exited with return code 0
  1362. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup-data:1_notify_0 from 781:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-237
  1363. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_notify_0 (call=281, rc=0, cib-update=0, confirmed=true) ok
  1364. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=301:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup-data:1_monitor_31000 )
  1365. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup-data:1 monitor[282] (pid 4304)
  1366. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=264:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_demote_0 )
  1367. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 demote[283] (pid 4305)
  1368. Nov 27 01:21:53 vs001 kernel: [44345.420882] block drbd31: role( Primary -> Secondary )
  1369. Nov 27 01:21:53 vs001 kernel: [44345.420898] block drbd31: bitmap WRITE of 0 pages took 0 jiffies
  1370. Nov 27 01:21:53 vs001 kernel: [44345.421117] block drbd31: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1371. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation monitor[282] on drbd_backup-data:1 for client 1016: pid 4304 exited with return code 0
  1372. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup-data:1_monitor_31000 (call=282, rc=0, cib-update=221, confirmed=false) ok
  1373. Nov 27 01:21:53 vs001 lrmd: [1013]: info: RA output: (drbd_backup:0:demote:stdout)
  1374. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation demote[283] on drbd_backup:0 for client 1016: pid 4305 exited with return code 0
  1375. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_demote_0 (call=283, rc=0, cib-update=222, confirmed=true) ok
  1376. Nov 27 01:21:53 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=763:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_notify_0 )
  1377. Nov 27 01:21:53 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 notify[284] (pid 4371)
  1378. Nov 27 01:21:53 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_backup:0 (1000)
  1379. Nov 27 01:21:53 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 605: master-drbd_backup:0=1000
  1380. Nov 27 01:21:53 vs001 lrmd: [1013]: info: RA output: (drbd_backup:0:notify:stdout)
  1381. Nov 27 01:21:53 vs001 lrmd: [1013]: info: operation notify[284] on drbd_backup:0 for client 1016: pid 4371 exited with return code 0
  1382. Nov 27 01:21:53 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_backup:0_notify_0 from 763:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511713-238
  1383. Nov 27 01:21:53 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_notify_0 (call=284, rc=0, cib-update=0, confirmed=true) ok
  1384. Nov 27 01:21:54 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[275] (pid 4399)
  1385. Nov 27 01:21:54 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[276] (pid 4400)
  1386. Nov 27 01:21:54 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[277] (pid 4401)
  1387. Nov 27 01:21:54 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[278] (pid 4402)
  1388. Nov 27 01:21:54 vs001 lrmd: [1013]: info: operation notify[275] on drbd_sheppard:0 for client 1016: pid 4399 exited with return code 0
  1389. Nov 27 01:21:54 vs001 lrmd: [1013]: info: operation notify[276] on drbd_sheppard-data:0 for client 1016: pid 4400 exited with return code 0
  1390. Nov 27 01:21:54 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 810:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511714-239
  1391. Nov 27 01:21:54 vs001 lrmd: [1013]: info: operation notify[277] on drbd_tealc:1 for client 1016: pid 4401 exited with return code 0
  1392. Nov 27 01:21:54 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=275, rc=0, cib-update=0, confirmed=true) ok
  1393. Nov 27 01:21:54 vs001 lrmd: [1013]: info: operation notify[278] on drbd_tealc-data:1 for client 1016: pid 4402 exited with return code 0
  1394. Nov 27 01:21:54 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_notify_0 from 826:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511714-240
  1395. Nov 27 01:21:54 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_notify_0 (call=276, rc=0, cib-update=0, confirmed=true) ok
  1396. Nov 27 01:21:54 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 844:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511714-241
  1397. Nov 27 01:21:54 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=277, rc=0, cib-update=0, confirmed=true) ok
  1398. Nov 27 01:21:54 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 860:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511714-242
  1399. Nov 27 01:21:54 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=278, rc=0, cib-update=0, confirmed=true) ok
  1400. Nov 27 01:21:55 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[279] (pid 4492)
  1401. Nov 27 01:21:55 vs001 lrmd: [1013]: info: operation notify[279] on drbd_cinder-HDD1:1 for client 1016: pid 4492 exited with return code 0
  1402. Nov 27 01:21:55 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 892:4857:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511715-243
  1403. Nov 27 01:21:55 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=279, rc=0, cib-update=0, confirmed=true) ok
  1404. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=266:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_backup:0_monitor_31000 )
  1405. Nov 27 01:21:59 vs001 lrmd: [1013]: info: rsc:drbd_backup:0 monitor[285] (pid 4538)
  1406. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=809:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1407. Nov 27 01:21:59 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[286] (pid 4539)
  1408. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=825:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_notify_0 )
  1409. Nov 27 01:21:59 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[287] (pid 4540)
  1410. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=843:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1411. Nov 27 01:21:59 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[288] (pid 4541)
  1412. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=859:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1413. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=891:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1414. Nov 27 01:21:59 vs001 lrmd: [1013]: info: operation notify[286] on drbd_sheppard:0 for client 1016: pid 4539 exited with return code 0
  1415. Nov 27 01:21:59 vs001 lrmd: [1013]: info: operation notify[287] on drbd_sheppard-data:0 for client 1016: pid 4540 exited with return code 0
  1416. Nov 27 01:21:59 vs001 lrmd: [1013]: info: operation notify[288] on drbd_tealc:1 for client 1016: pid 4541 exited with return code 0
  1417. Nov 27 01:21:59 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 809:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511719-244
  1418. Nov 27 01:21:59 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=286, rc=0, cib-update=0, confirmed=true) ok
  1419. Nov 27 01:21:59 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_notify_0 from 825:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511719-245
  1420. Nov 27 01:21:59 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_notify_0 (call=287, rc=0, cib-update=0, confirmed=true) ok
  1421. Nov 27 01:21:59 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 843:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511719-246
  1422. Nov 27 01:21:59 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=288, rc=0, cib-update=0, confirmed=true) ok
  1423. Nov 27 01:21:59 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=409:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_demote_0 )
  1424. Nov 27 01:21:59 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 demote[291] (pid 4637)
  1425. Nov 27 01:21:59 vs001 lrmd: [1013]: info: operation monitor[285] on drbd_backup:0 for client 1016: pid 4538 exited with return code 0
  1426. Nov 27 01:21:59 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_backup:0_monitor_31000 (call=285, rc=0, cib-update=223, confirmed=false) ok
  1427. Nov 27 01:21:59 vs001 kernel: [44351.717967] block drbd27: role( Primary -> Secondary )
  1428. Nov 27 01:21:59 vs001 kernel: [44351.717981] block drbd27: bitmap WRITE of 0 pages took 0 jiffies
  1429. Nov 27 01:21:59 vs001 kernel: [44351.747213] block drbd27: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1430. Nov 27 01:22:00 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard-data:0:demote:stdout)
  1431. Nov 27 01:22:00 vs001 lrmd: [1013]: info: operation demote[291] on drbd_sheppard-data:0 for client 1016: pid 4637 exited with return code 0
  1432. Nov 27 01:22:00 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_demote_0 (call=291, rc=0, cib-update=224, confirmed=true) ok
  1433. Nov 27 01:22:00 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=826:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_notify_0 )
  1434. Nov 27 01:22:00 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 notify[292] (pid 4675)
  1435. Nov 27 01:22:00 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard-data:0 (1000)
  1436. Nov 27 01:22:00 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 607: master-drbd_sheppard-data:0=1000
  1437. Nov 27 01:22:00 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard-data:0:notify:stdout)
  1438. Nov 27 01:22:00 vs001 lrmd: [1013]: info: operation notify[292] on drbd_sheppard-data:0 for client 1016: pid 4675 exited with return code 0
  1439. Nov 27 01:22:00 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard-data:0_notify_0 from 826:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511720-247
  1440. Nov 27 01:22:00 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_notify_0 (call=292, rc=0, cib-update=0, confirmed=true) ok
  1441. Nov 27 01:22:00 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[289] (pid 4704)
  1442. Nov 27 01:22:00 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[290] (pid 4705)
  1443. Nov 27 01:22:00 vs001 lrmd: [1013]: info: operation notify[290] on drbd_cinder-HDD1:1 for client 1016: pid 4705 exited with return code 0
  1444. Nov 27 01:22:00 vs001 lrmd: [1013]: info: operation notify[289] on drbd_tealc-data:1 for client 1016: pid 4704 exited with return code 0
  1445. Nov 27 01:22:00 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 891:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511720-248
  1446. Nov 27 01:22:00 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=290, rc=0, cib-update=0, confirmed=true) ok
  1447. Nov 27 01:22:00 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 859:4858:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511720-249
  1448. Nov 27 01:22:00 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=289, rc=0, cib-update=0, confirmed=true) ok
  1449. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=410:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard-data:0_monitor_31000 )
  1450. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_sheppard-data:0 monitor[293] (pid 4814)
  1451. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=810:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1452. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[294] (pid 4815)
  1453. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=844:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1454. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[295] (pid 4816)
  1455. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=860:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1456. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[296] (pid 4817)
  1457. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=892:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1458. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation notify[295] on drbd_tealc:1 for client 1016: pid 4816 exited with return code 0
  1459. Nov 27 01:22:13 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 844:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511733-250
  1460. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=295, rc=0, cib-update=0, confirmed=true) ok
  1461. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation notify[294] on drbd_sheppard:0 for client 1016: pid 4815 exited with return code 0
  1462. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation notify[296] on drbd_tealc-data:1 for client 1016: pid 4817 exited with return code 0
  1463. Nov 27 01:22:13 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 810:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511733-251
  1464. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=294, rc=0, cib-update=0, confirmed=true) ok
  1465. Nov 27 01:22:13 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 860:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511733-252
  1466. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=296, rc=0, cib-update=0, confirmed=true) ok
  1467. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=485:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_demote_0 )
  1468. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 demote[298] (pid 4912)
  1469. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=377:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_demote_0 )
  1470. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 demote[299] (pid 4913)
  1471. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation monitor[293] on drbd_sheppard-data:0 for client 1016: pid 4814 exited with return code 0
  1472. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard-data:0_monitor_31000 (call=293, rc=0, cib-update=225, confirmed=false) ok
  1473. Nov 27 01:22:13 vs001 kernel: [44365.235137] block drbd21: role( Primary -> Secondary )
  1474. Nov 27 01:22:13 vs001 kernel: [44365.235158] block drbd21: bitmap WRITE of 0 pages took 0 jiffies
  1475. Nov 27 01:22:13 vs001 kernel: [44365.236184] block drbd30: role( Primary -> Secondary )
  1476. Nov 27 01:22:13 vs001 kernel: [44365.236198] block drbd30: bitmap WRITE of 0 pages took 0 jiffies
  1477. Nov 27 01:22:13 vs001 kernel: [44365.292187] block drbd21: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1478. Nov 27 01:22:13 vs001 kernel: [44365.292292] block drbd30: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1479. Nov 27 01:22:13 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard:0:demote:stdout)
  1480. Nov 27 01:22:13 vs001 lrmd: [1013]: info: RA output: (drbd_tealc-data:1:demote:stdout)
  1481. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation demote[299] on drbd_sheppard:0 for client 1016: pid 4913 exited with return code 0
  1482. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_demote_0 (call=299, rc=0, cib-update=226, confirmed=true) ok
  1483. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation demote[298] on drbd_tealc-data:1 for client 1016: pid 4912 exited with return code 0
  1484. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_demote_0 (call=298, rc=0, cib-update=227, confirmed=true) ok
  1485. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=811:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_notify_0 )
  1486. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 notify[300] (pid 5061)
  1487. Nov 27 01:22:13 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=861:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_notify_0 )
  1488. Nov 27 01:22:13 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 notify[301] (pid 5062)
  1489. Nov 27 01:22:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc-data:1 (1000)
  1490. Nov 27 01:22:13 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 610: master-drbd_tealc-data:1=1000
  1491. Nov 27 01:22:13 vs001 lrmd: [1013]: info: RA output: (drbd_tealc-data:1:notify:stdout)
  1492. Nov 27 01:22:13 vs001 lrmd: [1013]: info: RA output: (drbd_sheppard:0:notify:stdout)
  1493. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation notify[301] on drbd_tealc-data:1 for client 1016: pid 5062 exited with return code 0
  1494. Nov 27 01:22:13 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_sheppard:0 (1000)
  1495. Nov 27 01:22:13 vs001 lrmd: [1013]: info: operation notify[300] on drbd_sheppard:0 for client 1016: pid 5061 exited with return code 0
  1496. Nov 27 01:22:13 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc-data:1_notify_0 from 861:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511733-253
  1497. Nov 27 01:22:13 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 612: master-drbd_sheppard:0=1000
  1498. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_notify_0 (call=301, rc=0, cib-update=0, confirmed=true) ok
  1499. Nov 27 01:22:13 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_sheppard:0_notify_0 from 811:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511733-254
  1500. Nov 27 01:22:13 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_notify_0 (call=300, rc=0, cib-update=0, confirmed=true) ok
  1501. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[297] (pid 5117)
  1502. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation notify[297] on drbd_cinder-HDD1:1 for client 1016: pid 5117 exited with return code 0
  1503. Nov 27 01:22:14 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 892:4859:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511734-255
  1504. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=297, rc=0, cib-update=0, confirmed=true) ok
  1505. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=381:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_sheppard:0_monitor_31000 )
  1506. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_sheppard:0 monitor[302] (pid 5140)
  1507. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=488:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc-data:1_monitor_31000 )
  1508. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_tealc-data:1 monitor[303] (pid 5141)
  1509. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=846:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1510. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[304] (pid 5143)
  1511. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=894:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1512. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[305] (pid 5147)
  1513. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation notify[305] on drbd_cinder-HDD1:1 for client 1016: pid 5147 exited with return code 0
  1514. Nov 27 01:22:14 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 894:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511734-256
  1515. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=305, rc=0, cib-update=0, confirmed=true) ok
  1516. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation notify[304] on drbd_tealc:1 for client 1016: pid 5143 exited with return code 0
  1517. Nov 27 01:22:14 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 846:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511734-257
  1518. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=304, rc=0, cib-update=0, confirmed=true) ok
  1519. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=455:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_demote_0 )
  1520. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 demote[306] (pid 5243)
  1521. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation monitor[302] on drbd_sheppard:0 for client 1016: pid 5140 exited with return code 0
  1522. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation monitor[303] on drbd_tealc-data:1 for client 1016: pid 5141 exited with return code 0
  1523. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_sheppard:0_monitor_31000 (call=302, rc=0, cib-update=228, confirmed=false) ok
  1524. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc-data:1_monitor_31000 (call=303, rc=0, cib-update=229, confirmed=false) ok
  1525. Nov 27 01:22:14 vs001 kernel: [44366.380332] block drbd20: role( Primary -> Secondary )
  1526. Nov 27 01:22:14 vs001 kernel: [44366.380345] block drbd20: bitmap WRITE of 0 pages took 0 jiffies
  1527. Nov 27 01:22:14 vs001 kernel: [44366.410574] block drbd20: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1528. Nov 27 01:22:14 vs001 lrmd: [1013]: info: RA output: (drbd_tealc:1:demote:stdout)
  1529. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation demote[306] on drbd_tealc:1 for client 1016: pid 5243 exited with return code 0
  1530. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_demote_0 (call=306, rc=0, cib-update=230, confirmed=true) ok
  1531. Nov 27 01:22:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=847:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_notify_0 )
  1532. Nov 27 01:22:14 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 notify[307] (pid 5278)
  1533. Nov 27 01:22:14 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_tealc:1 (1000)
  1534. Nov 27 01:22:14 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 614: master-drbd_tealc:1=1000
  1535. Nov 27 01:22:14 vs001 lrmd: [1013]: info: RA output: (drbd_tealc:1:notify:stdout)
  1536. Nov 27 01:22:14 vs001 lrmd: [1013]: info: operation notify[307] on drbd_tealc:1 for client 1016: pid 5278 exited with return code 0
  1537. Nov 27 01:22:14 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_tealc:1_notify_0 from 847:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511734-258
  1538. Nov 27 01:22:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_notify_0 (call=307, rc=0, cib-update=0, confirmed=true) ok
  1539. Nov 27 01:22:34 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=895:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1540. Nov 27 01:22:34 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[308] (pid 5594)
  1541. Nov 27 01:22:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:notify:stdout)
  1542. Nov 27 01:22:34 vs001 lrmd: [1013]: info: operation notify[308] on drbd_cinder-HDD1:1 for client 1016: pid 5594 exited with return code 0
  1543. Nov 27 01:22:34 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 895:4860:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511754-259
  1544. Nov 27 01:22:34 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=308, rc=0, cib-update=0, confirmed=true) ok
  1545. Nov 27 01:22:34 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=458:4861:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_tealc:1_monitor_31000 )
  1546. Nov 27 01:22:34 vs001 lrmd: [1013]: info: rsc:drbd_tealc:1 monitor[309] (pid 5622)
  1547. Nov 27 01:22:34 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=882:4861:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1548. Nov 27 01:22:34 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[310] (pid 5623)
  1549. Nov 27 01:22:34 vs001 lrmd: [1013]: info: operation notify[310] on drbd_cinder-HDD1:1 for client 1016: pid 5623 exited with return code 0
  1550. Nov 27 01:22:34 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 882:4861:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511754-260
  1551. Nov 27 01:22:34 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=310, rc=0, cib-update=0, confirmed=true) ok
  1552. Nov 27 01:22:34 vs001 lrmd: [1013]: info: operation monitor[309] on drbd_tealc:1 for client 1016: pid 5622 exited with return code 0
  1553. Nov 27 01:22:34 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_tealc:1_monitor_31000 (call=309, rc=0, cib-update=231, confirmed=false) ok
  1554. Nov 27 01:23:11 vs001 cib: [1012]: info: cib_stats: Processed 808 operations (1250.00us average, 0% utilization) in the last 10min
  1555. Nov 27 01:24:14 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=883:4861:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1556. Nov 27 01:24:14 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[311] (pid 7148)
  1557. Nov 27 01:24:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:notify:stdout)
  1558. Nov 27 01:24:14 vs001 lrmd: [1013]: info: operation notify[311] on drbd_cinder-HDD1:1 for client 1016: pid 7148 exited with return code 0
  1559. Nov 27 01:24:14 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 883:4861:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511854-261
  1560. Nov 27 01:24:14 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=311, rc=0, cib-update=0, confirmed=true) ok
  1561. Nov 27 01:24:15 vs001 lrmd: [1013]: info: cancel_op: operation monitor[229] on drbd_cinder-HDD1:1 for client 1016, its parameters: drbd_resource=[cinder-HDD1] CRM_meta_role=[Master] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_start_resource=[drbd_cinder-HDD1:0 drbd_cinder-HDD1:1 ] CRM_meta_notify_inactive_resource=[drbd_cinder-HDD1:0 drbd_cinder-HDD1:1 ] CRM_meta_notify_master_uname=[ ] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.5] CRM_meta_notify=[true] CR cancelled
  1562. Nov 27 01:24:15 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_monitor_29000 from 10:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511855-262
  1563. Nov 27 01:24:15 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=895:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1564. Nov 27 01:24:15 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[312] (pid 7176)
  1565. Nov 27 01:24:15 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_monitor_29000 (call=229, status=1, cib-update=0, confirmed=true) Cancelled
  1566. Nov 27 01:24:15 vs001 lrmd: [1013]: info: operation notify[312] on drbd_cinder-HDD1:1 for client 1016: pid 7176 exited with return code 0
  1567. Nov 27 01:24:15 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 895:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511855-263
  1568. Nov 27 01:24:15 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=312, rc=0, cib-update=0, confirmed=true) ok
  1569. Nov 27 01:24:15 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=571:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_demote_0 )
  1570. Nov 27 01:24:15 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 demote[313] (pid 7199)
  1571. Nov 27 01:24:15 vs001 kernel: [44486.860319] block drbd16: State change failed: Device is held open by someone
  1572. Nov 27 01:24:15 vs001 kernel: [44486.861531] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1573. Nov 27 01:24:15 vs001 kernel: [44486.862736] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1574. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1575. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1576. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1577. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1578. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1579. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1580. Nov 27 01:24:15 vs001 kernel: [44486.883742] block drbd16: State change failed: Device is held open by someone
  1581. Nov 27 01:24:15 vs001 kernel: [44486.884987] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1582. Nov 27 01:24:15 vs001 kernel: [44486.886239] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1583. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1584. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1585. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1586. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1587. Nov 27 01:24:15 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1588. Nov 27 01:24:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1589. Nov 27 01:24:16 vs001 kernel: [44487.909574] block drbd16: State change failed: Device is held open by someone
  1590. Nov 27 01:24:16 vs001 kernel: [44487.910869] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1591. Nov 27 01:24:16 vs001 kernel: [44487.912188] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1592. Nov 27 01:24:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1593. Nov 27 01:24:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1594. Nov 27 01:24:16 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1595. Nov 27 01:24:16 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1596. Nov 27 01:24:16 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1597. Nov 27 01:24:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1598. Nov 27 01:24:17 vs001 kernel: [44488.934385] block drbd16: State change failed: Device is held open by someone
  1599. Nov 27 01:24:17 vs001 kernel: [44488.935748] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1600. Nov 27 01:24:17 vs001 kernel: [44488.937161] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1601. Nov 27 01:24:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1602. Nov 27 01:24:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1603. Nov 27 01:24:17 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1604. Nov 27 01:24:17 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1605. Nov 27 01:24:17 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1606. Nov 27 01:24:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1607. Nov 27 01:24:18 vs001 kernel: [44489.959531] block drbd16: State change failed: Device is held open by someone
  1608. Nov 27 01:24:18 vs001 kernel: [44489.960969] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1609. Nov 27 01:24:18 vs001 kernel: [44489.962416] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1610. Nov 27 01:24:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone#012Command 'drbdsetup 16 secondary
  1611. Nov 27 01:24:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) ' terminated with exit code 11
  1612. Nov 27 01:24:18 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1613. Nov 27 01:24:18 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1614. Nov 27 01:24:18 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1615. Nov 27 01:24:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1616. Nov 27 01:24:19 vs001 kernel: [44490.984960] block drbd16: State change failed: Device is held open by someone
  1617. Nov 27 01:24:19 vs001 kernel: [44490.986448] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1618. Nov 27 01:24:19 vs001 kernel: [44490.987932] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1619. Nov 27 01:24:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1620. Nov 27 01:24:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1621. Nov 27 01:24:19 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1622. Nov 27 01:24:19 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1623. Nov 27 01:24:19 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1624. Nov 27 01:24:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1625. Nov 27 01:24:20 vs001 kernel: [44492.010295] block drbd16: State change failed: Device is held open by someone
  1626. Nov 27 01:24:20 vs001 kernel: [44492.011807] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1627. Nov 27 01:24:20 vs001 kernel: [44492.013328] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1628. Nov 27 01:24:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1629. Nov 27 01:24:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1630. Nov 27 01:24:20 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1631. Nov 27 01:24:20 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1632. Nov 27 01:24:20 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1633. Nov 27 01:24:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1634. Nov 27 01:24:21 vs001 kernel: [44493.036031] block drbd16: State change failed: Device is held open by someone
  1635. Nov 27 01:24:21 vs001 kernel: [44493.037578] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1636. Nov 27 01:24:21 vs001 kernel: [44493.039107] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1637. Nov 27 01:24:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1638. Nov 27 01:24:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1639. Nov 27 01:24:21 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1640. Nov 27 01:24:21 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1641. Nov 27 01:24:21 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1642. Nov 27 01:24:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1643. Nov 27 01:24:22 vs001 kernel: [44494.061904] block drbd16: State change failed: Device is held open by someone
  1644. Nov 27 01:24:22 vs001 kernel: [44494.063442] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1645. Nov 27 01:24:22 vs001 kernel: [44494.065001] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1646. Nov 27 01:24:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1647. Nov 27 01:24:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1648. Nov 27 01:24:22 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1649. Nov 27 01:24:22 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1650. Nov 27 01:24:22 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1651. Nov 27 01:24:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1652. Nov 27 01:24:23 vs001 kernel: [44495.087747] block drbd16: State change failed: Device is held open by someone
  1653. Nov 27 01:24:23 vs001 kernel: [44495.089334] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1654. Nov 27 01:24:23 vs001 kernel: [44495.090916] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1655. Nov 27 01:24:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1656. Nov 27 01:24:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1657. Nov 27 01:24:23 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1658. Nov 27 01:24:23 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1659. Nov 27 01:24:23 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1660. Nov 27 01:24:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1661. Nov 27 01:24:24 vs001 kernel: [44496.113672] block drbd16: State change failed: Device is held open by someone
  1662. Nov 27 01:24:24 vs001 kernel: [44496.115266] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1663. Nov 27 01:24:24 vs001 kernel: [44496.116889] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1664. Nov 27 01:24:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1665. Nov 27 01:24:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1666. Nov 27 01:24:24 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1667. Nov 27 01:24:24 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1668. Nov 27 01:24:24 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1669. Nov 27 01:24:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1670. Nov 27 01:24:25 vs001 kernel: [44497.139651] block drbd16: State change failed: Device is held open by someone
  1671. Nov 27 01:24:25 vs001 kernel: [44497.141271] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1672. Nov 27 01:24:25 vs001 kernel: [44497.142890] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1673. Nov 27 01:24:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1674. Nov 27 01:24:25 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1675. Nov 27 01:24:25 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1676. Nov 27 01:24:25 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1677. Nov 27 01:24:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1678. Nov 27 01:24:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1679. Nov 27 01:24:26 vs001 kernel: [44498.165345] block drbd16: State change failed: Device is held open by someone
  1680. Nov 27 01:24:26 vs001 kernel: [44498.166971] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1681. Nov 27 01:24:26 vs001 kernel: [44498.168616] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1682. Nov 27 01:24:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1683. Nov 27 01:24:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1684. Nov 27 01:24:26 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1685. Nov 27 01:24:26 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1686. Nov 27 01:24:26 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1687. Nov 27 01:24:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1688. Nov 27 01:24:27 vs001 kernel: [44499.191761] block drbd16: State change failed: Device is held open by someone
  1689. Nov 27 01:24:27 vs001 kernel: [44499.193422] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1690. Nov 27 01:24:27 vs001 kernel: [44499.195087] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1691. Nov 27 01:24:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1692. Nov 27 01:24:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command '
  1693. Nov 27 01:24:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) drbdsetup 16 secondary' terminated with exit code 11
  1694. Nov 27 01:24:27 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1695. Nov 27 01:24:27 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1696. Nov 27 01:24:27 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1697. Nov 27 01:24:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1698. Nov 27 01:24:28 vs001 kernel: [44500.218230] block drbd16: State change failed: Device is held open by someone
  1699. Nov 27 01:24:28 vs001 kernel: [44500.219910] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1700. Nov 27 01:24:28 vs001 kernel: [44500.221616] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1701. Nov 27 01:24:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1702. Nov 27 01:24:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary
  1703. Nov 27 01:24:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) ' terminated with exit code 11
  1704. Nov 27 01:24:28 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1705. Nov 27 01:24:28 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1706. Nov 27 01:24:28 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1707. Nov 27 01:24:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1708. Nov 27 01:24:29 vs001 kernel: [44501.244523] block drbd16: State change failed: Device is held open by someone
  1709. Nov 27 01:24:29 vs001 kernel: [44501.246225] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1710. Nov 27 01:24:29 vs001 kernel: [44501.247944] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1711. Nov 27 01:24:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1712. Nov 27 01:24:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1713. Nov 27 01:24:29 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1714. Nov 27 01:24:29 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1715. Nov 27 01:24:29 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1716. Nov 27 01:24:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1717. Nov 27 01:24:30 vs001 kernel: [44502.271030] block drbd16: State change failed: Device is held open by someone
  1718. Nov 27 01:24:30 vs001 kernel: [44502.272761] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1719. Nov 27 01:24:30 vs001 kernel: [44502.274508] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1720. Nov 27 01:24:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1721. Nov 27 01:24:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1722. Nov 27 01:24:30 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1723. Nov 27 01:24:30 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1724. Nov 27 01:24:30 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1725. Nov 27 01:24:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1726. Nov 27 01:24:31 vs001 kernel: [44503.298728] block drbd16: State change failed: Device is held open by someone
  1727. Nov 27 01:24:31 vs001 kernel: [44503.300595] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1728. Nov 27 01:24:31 vs001 kernel: [44503.302333] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1729. Nov 27 01:24:31 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1730. Nov 27 01:24:31 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary
  1731. Nov 27 01:24:31 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) ' terminated with exit code 11
  1732. Nov 27 01:24:31 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1733. Nov 27 01:24:31 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1734. Nov 27 01:24:31 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1735. Nov 27 01:24:31 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1736. Nov 27 01:24:32 vs001 kernel: [44504.324835] block drbd16: State change failed: Device is held open by someone
  1737. Nov 27 01:24:32 vs001 kernel: [44504.326555] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1738. Nov 27 01:24:32 vs001 kernel: [44504.328277] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1739. Nov 27 01:24:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1740. Nov 27 01:24:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1741. Nov 27 01:24:32 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1742. Nov 27 01:24:32 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1743. Nov 27 01:24:32 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1744. Nov 27 01:24:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1745. Nov 27 01:24:33 vs001 kernel: [44505.349537] block drbd16: State change failed: Device is held open by someone
  1746. Nov 27 01:24:33 vs001 kernel: [44505.351583] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1747. Nov 27 01:24:33 vs001 kernel: [44505.353946] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1748. Nov 27 01:24:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1749. Nov 27 01:24:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1750. Nov 27 01:24:33 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1751. Nov 27 01:24:33 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1752. Nov 27 01:24:33 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1753. Nov 27 01:24:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1754. Nov 27 01:24:34 vs001 kernel: [44506.376286] block drbd16: State change failed: Device is held open by someone
  1755. Nov 27 01:24:34 vs001 kernel: [44506.378001] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1756. Nov 27 01:24:34 vs001 kernel: [44506.379722] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1757. Nov 27 01:24:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) 16: State change failed: (-12) Device is held open by someone
  1758. Nov 27 01:24:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1759. Nov 27 01:24:34 vs001 drbd[7199]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1760. Nov 27 01:24:34 vs001 drbd[7199]: ERROR: cinder-HDD1: Exit code 11
  1761. Nov 27 01:24:34 vs001 drbd[7199]: ERROR: cinder-HDD1: Command output:
  1762. Nov 27 01:24:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:demote:stdout)
  1763. Nov 27 01:24:35 vs001 lrmd: [1013]: WARN: drbd_cinder-HDD1:1:demote process (PID 7199) timed out (try 1). Killing with signal SIGTERM (15).
  1764. Nov 27 01:24:35 vs001 lrmd: [1013]: WARN: operation demote[313] on drbd_cinder-HDD1:1 for client 1016: pid 7199 timed out
  1765. Nov 27 01:24:35 vs001 crmd: [1016]: ERROR: process_lrm_event: LRM operation drbd_cinder-HDD1:1_demote_0 (313) Timed Out (timeout=20000ms)
  1766. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  1767. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_cinder-HDD1:1 (1)
  1768. Nov 27 01:24:35 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=896:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1769. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 621: fail-count-drbd_cinder-HDD1:1=1
  1770. Nov 27 01:24:35 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[314] (pid 7920)
  1771. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  1772. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_cinder-HDD1:1 (1385511875)
  1773. Nov 27 01:24:35 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 624: last-failure-drbd_cinder-HDD1:1=1385511875
  1774. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:notify:stdout)
  1775. Nov 27 01:24:35 vs001 lrmd: [1013]: info: operation notify[314] on drbd_cinder-HDD1:1 for client 1016: pid 7920 exited with return code 0
  1776. Nov 27 01:24:35 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 896:4862:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511875-264
  1777. Nov 27 01:24:35 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=314, rc=0, cib-update=0, confirmed=true) ok
  1778. Nov 27 01:24:35 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=883:4863:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_notify_0 )
  1779. Nov 27 01:24:35 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 notify[315] (pid 7976)
  1780. Nov 27 01:24:35 vs001 lrmd: [1013]: info: operation notify[315] on drbd_cinder-HDD1:1 for client 1016: pid 7976 exited with return code 0
  1781. Nov 27 01:24:35 vs001 crmd: [1016]: info: send_direct_ack: ACK'ing resource op drbd_cinder-HDD1:1_notify_0 from 883:4863:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452: lrm_invoke-lrmd-1385511875-265
  1782. Nov 27 01:24:35 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_notify_0 (call=315, rc=0, cib-update=0, confirmed=true) ok
  1783. Nov 27 01:24:35 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=10:4863:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_stop_0 )
  1784. Nov 27 01:24:35 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 stop[316] (pid 7999)
  1785. Nov 27 01:24:35 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1786. Nov 27 01:24:35 vs001 kernel: [44507.102987] block drbd16: State change failed: Device is held open by someone
  1787. Nov 27 01:24:35 vs001 kernel: [44507.104707] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1788. Nov 27 01:24:35 vs001 kernel: [44507.106431] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1789. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1790. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16
  1791. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) secondary' terminated with exit code 11
  1792. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1793. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1794. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1795. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1796. Nov 27 01:24:35 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1797. Nov 27 01:24:35 vs001 kernel: [44507.131052] block drbd16: State change failed: Device is held open by someone
  1798. Nov 27 01:24:35 vs001 kernel: [44507.132768] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1799. Nov 27 01:24:35 vs001 kernel: [44507.134488] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1800. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1801. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1802. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1803. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1804. Nov 27 01:24:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1805. Nov 27 01:24:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1806. Nov 27 01:24:36 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1807. Nov 27 01:24:36 vs001 kernel: [44508.160433] block drbd16: State change failed: Device is held open by someone
  1808. Nov 27 01:24:36 vs001 kernel: [44508.162156] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1809. Nov 27 01:24:36 vs001 kernel: [44508.163877] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1810. Nov 27 01:24:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1811. Nov 27 01:24:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1812. Nov 27 01:24:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1813. Nov 27 01:24:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1814. Nov 27 01:24:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1815. Nov 27 01:24:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1816. Nov 27 01:24:37 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1817. Nov 27 01:24:37 vs001 kernel: [44509.190914] block drbd16: State change failed: Device is held open by someone
  1818. Nov 27 01:24:37 vs001 kernel: [44509.192635] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1819. Nov 27 01:24:37 vs001 kernel: [44509.194351] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1820. Nov 27 01:24:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone#012Command 'drbdsetup 16 secondary
  1821. Nov 27 01:24:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  1822. Nov 27 01:24:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1823. Nov 27 01:24:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1824. Nov 27 01:24:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1825. Nov 27 01:24:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1826. Nov 27 01:24:38 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1827. Nov 27 01:24:38 vs001 kernel: [44510.221923] block drbd16: State change failed: Device is held open by someone
  1828. Nov 27 01:24:38 vs001 kernel: [44510.223644] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1829. Nov 27 01:24:38 vs001 kernel: [44510.225373] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1830. Nov 27 01:24:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1831. Nov 27 01:24:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1832. Nov 27 01:24:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1833. Nov 27 01:24:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1834. Nov 27 01:24:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1835. Nov 27 01:24:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1836. Nov 27 01:24:39 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1837. Nov 27 01:24:39 vs001 kernel: [44511.252425] block drbd16: State change failed: Device is held open by someone
  1838. Nov 27 01:24:39 vs001 kernel: [44511.254136] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1839. Nov 27 01:24:39 vs001 kernel: [44511.255853] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1840. Nov 27 01:24:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1841. Nov 27 01:24:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1842. Nov 27 01:24:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1843. Nov 27 01:24:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1844. Nov 27 01:24:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1845. Nov 27 01:24:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1846. Nov 27 01:24:40 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1847. Nov 27 01:24:40 vs001 kernel: [44512.282871] block drbd16: State change failed: Device is held open by someone
  1848. Nov 27 01:24:40 vs001 kernel: [44512.284605] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1849. Nov 27 01:24:40 vs001 kernel: [44512.286335] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1850. Nov 27 01:24:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1851. Nov 27 01:24:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1852. Nov 27 01:24:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1853. Nov 27 01:24:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1854. Nov 27 01:24:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1855. Nov 27 01:24:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1856. Nov 27 01:24:41 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1857. Nov 27 01:24:41 vs001 kernel: [44513.313527] block drbd16: State change failed: Device is held open by someone
  1858. Nov 27 01:24:41 vs001 kernel: [44513.315838] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1859. Nov 27 01:24:41 vs001 kernel: [44513.318168] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1860. Nov 27 01:24:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1861. Nov 27 01:24:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  1862. Nov 27 01:24:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  1863. Nov 27 01:24:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1864. Nov 27 01:24:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1865. Nov 27 01:24:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1866. Nov 27 01:24:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1867. Nov 27 01:24:42 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1868. Nov 27 01:24:42 vs001 kernel: [44514.338505] block drbd16: State change failed: Device is held open by someone
  1869. Nov 27 01:24:42 vs001 kernel: [44514.340227] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1870. Nov 27 01:24:42 vs001 kernel: [44514.341948] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1871. Nov 27 01:24:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1872. Nov 27 01:24:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  1873. Nov 27 01:24:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  1874. Nov 27 01:24:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1875. Nov 27 01:24:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1876. Nov 27 01:24:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1877. Nov 27 01:24:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1878. Nov 27 01:24:43 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1879. Nov 27 01:24:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1880. Nov 27 01:24:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1881. Nov 27 01:24:43 vs001 kernel: [44515.368246] block drbd16: State change failed: Device is held open by someone
  1882. Nov 27 01:24:43 vs001 kernel: [44515.369116] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1883. Nov 27 01:24:43 vs001 kernel: [44515.369986] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1884. Nov 27 01:24:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1885. Nov 27 01:24:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1886. Nov 27 01:24:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1887. Nov 27 01:24:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1888. Nov 27 01:24:44 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1889. Nov 27 01:24:44 vs001 kernel: [44516.388861] block drbd16: State change failed: Device is held open by someone
  1890. Nov 27 01:24:44 vs001 kernel: [44516.390572] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1891. Nov 27 01:24:44 vs001 kernel: [44516.392289] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1892. Nov 27 01:24:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1893. Nov 27 01:24:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1894. Nov 27 01:24:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1895. Nov 27 01:24:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1896. Nov 27 01:24:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1897. Nov 27 01:24:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1898. Nov 27 01:24:45 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1899. Nov 27 01:24:45 vs001 kernel: [44517.419186] block drbd16: State change failed: Device is held open by someone
  1900. Nov 27 01:24:45 vs001 kernel: [44517.420905] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1901. Nov 27 01:24:45 vs001 kernel: [44517.422619] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1902. Nov 27 01:24:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1903. Nov 27 01:24:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1904. Nov 27 01:24:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1905. Nov 27 01:24:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1906. Nov 27 01:24:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1907. Nov 27 01:24:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1908. Nov 27 01:24:46 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1909. Nov 27 01:24:46 vs001 kernel: [44518.449622] block drbd16: State change failed: Device is held open by someone
  1910. Nov 27 01:24:46 vs001 kernel: [44518.451337] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1911. Nov 27 01:24:46 vs001 kernel: [44518.453060] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1912. Nov 27 01:24:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1913. Nov 27 01:24:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1914. Nov 27 01:24:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1915. Nov 27 01:24:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1916. Nov 27 01:24:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1917. Nov 27 01:24:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1918. Nov 27 01:24:47 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1919. Nov 27 01:24:47 vs001 kernel: [44519.480419] block drbd16: State change failed: Device is held open by someone
  1920. Nov 27 01:24:47 vs001 kernel: [44519.482131] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1921. Nov 27 01:24:47 vs001 kernel: [44519.483846] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1922. Nov 27 01:24:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1923. Nov 27 01:24:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1924. Nov 27 01:24:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1925. Nov 27 01:24:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1926. Nov 27 01:24:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1927. Nov 27 01:24:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1928. Nov 27 01:24:48 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1929. Nov 27 01:24:48 vs001 kernel: [44520.510984] block drbd16: State change failed: Device is held open by someone
  1930. Nov 27 01:24:48 vs001 kernel: [44520.512705] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1931. Nov 27 01:24:48 vs001 kernel: [44520.514421] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1932. Nov 27 01:24:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1933. Nov 27 01:24:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1934. Nov 27 01:24:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1935. Nov 27 01:24:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1936. Nov 27 01:24:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1937. Nov 27 01:24:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1938. Nov 27 01:24:49 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1939. Nov 27 01:24:49 vs001 kernel: [44521.541538] block drbd16: State change failed: Device is held open by someone
  1940. Nov 27 01:24:49 vs001 kernel: [44521.543618] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1941. Nov 27 01:24:49 vs001 kernel: [44521.545515] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1942. Nov 27 01:24:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1943. Nov 27 01:24:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1944. Nov 27 01:24:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1945. Nov 27 01:24:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1946. Nov 27 01:24:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1947. Nov 27 01:24:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1948. Nov 27 01:24:50 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1949. Nov 27 01:24:50 vs001 kernel: [44522.574963] block drbd16: State change failed: Device is held open by someone
  1950. Nov 27 01:24:50 vs001 kernel: [44522.576682] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1951. Nov 27 01:24:50 vs001 kernel: [44522.578400] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1952. Nov 27 01:24:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1953. Nov 27 01:24:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1954. Nov 27 01:24:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1955. Nov 27 01:24:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1956. Nov 27 01:24:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1957. Nov 27 01:24:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1958. Nov 27 01:24:51 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1959. Nov 27 01:24:51 vs001 kernel: [44523.604756] block drbd16: State change failed: Device is held open by someone
  1960. Nov 27 01:24:51 vs001 kernel: [44523.606473] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1961. Nov 27 01:24:51 vs001 kernel: [44523.608213] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1962. Nov 27 01:24:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1963. Nov 27 01:24:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1964. Nov 27 01:24:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1965. Nov 27 01:24:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1966. Nov 27 01:24:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1967. Nov 27 01:24:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1968. Nov 27 01:24:52 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1969. Nov 27 01:24:52 vs001 kernel: [44524.635314] block drbd16: State change failed: Device is held open by someone
  1970. Nov 27 01:24:52 vs001 kernel: [44524.637039] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1971. Nov 27 01:24:52 vs001 kernel: [44524.638762] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1972. Nov 27 01:24:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1973. Nov 27 01:24:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1974. Nov 27 01:24:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1975. Nov 27 01:24:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1976. Nov 27 01:24:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1977. Nov 27 01:24:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1978. Nov 27 01:24:53 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1979. Nov 27 01:24:53 vs001 kernel: [44525.666004] block drbd16: State change failed: Device is held open by someone
  1980. Nov 27 01:24:53 vs001 kernel: [44525.667721] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1981. Nov 27 01:24:53 vs001 kernel: [44525.669450] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1982. Nov 27 01:24:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1983. Nov 27 01:24:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1984. Nov 27 01:24:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1985. Nov 27 01:24:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1986. Nov 27 01:24:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1987. Nov 27 01:24:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1988. Nov 27 01:24:54 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1989. Nov 27 01:24:54 vs001 kernel: [44526.696604] block drbd16: State change failed: Device is held open by someone
  1990. Nov 27 01:24:54 vs001 kernel: [44526.698323] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  1991. Nov 27 01:24:54 vs001 kernel: [44526.700052] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  1992. Nov 27 01:24:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  1993. Nov 27 01:24:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  1994. Nov 27 01:24:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  1995. Nov 27 01:24:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  1996. Nov 27 01:24:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  1997. Nov 27 01:24:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  1998. Nov 27 01:24:55 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  1999. Nov 27 01:24:55 vs001 kernel: [44527.727323] block drbd16: State change failed: Device is held open by someone
  2000. Nov 27 01:24:55 vs001 kernel: [44527.729053] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2001. Nov 27 01:24:55 vs001 kernel: [44527.730773] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2002. Nov 27 01:24:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2003. Nov 27 01:24:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2004. Nov 27 01:24:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2005. Nov 27 01:24:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2006. Nov 27 01:24:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2007. Nov 27 01:24:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2008. Nov 27 01:24:56 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2009. Nov 27 01:24:56 vs001 kernel: [44528.757565] block drbd16: State change failed: Device is held open by someone
  2010. Nov 27 01:24:56 vs001 kernel: [44528.759279] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2011. Nov 27 01:24:56 vs001 kernel: [44528.761004] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2012. Nov 27 01:24:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2013. Nov 27 01:24:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2014. Nov 27 01:24:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2015. Nov 27 01:24:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2016. Nov 27 01:24:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2017. Nov 27 01:24:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2018. Nov 27 01:24:58 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2019. Nov 27 01:24:58 vs001 kernel: [44529.788029] block drbd16: State change failed: Device is held open by someone
  2020. Nov 27 01:24:58 vs001 kernel: [44529.789741] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2021. Nov 27 01:24:58 vs001 kernel: [44529.791458] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2022. Nov 27 01:24:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2023. Nov 27 01:24:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2024. Nov 27 01:24:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2025. Nov 27 01:24:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2026. Nov 27 01:24:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2027. Nov 27 01:24:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2028. Nov 27 01:24:59 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2029. Nov 27 01:24:59 vs001 kernel: [44530.818479] block drbd16: State change failed: Device is held open by someone
  2030. Nov 27 01:24:59 vs001 kernel: [44530.820198] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2031. Nov 27 01:24:59 vs001 kernel: [44530.821915] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2032. Nov 27 01:24:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2033. Nov 27 01:24:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2034. Nov 27 01:24:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2035. Nov 27 01:24:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2036. Nov 27 01:24:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2037. Nov 27 01:24:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2038. Nov 27 01:25:00 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2039. Nov 27 01:25:00 vs001 kernel: [44531.849026] block drbd16: State change failed: Device is held open by someone
  2040. Nov 27 01:25:00 vs001 kernel: [44531.850739] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2041. Nov 27 01:25:00 vs001 kernel: [44531.852468] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2042. Nov 27 01:25:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2043. Nov 27 01:25:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2044. Nov 27 01:25:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2045. Nov 27 01:25:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2046. Nov 27 01:25:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2047. Nov 27 01:25:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2048. Nov 27 01:25:01 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2049. Nov 27 01:25:01 vs001 kernel: [44532.879731] block drbd16: State change failed: Device is held open by someone
  2050. Nov 27 01:25:01 vs001 kernel: [44532.881446] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2051. Nov 27 01:25:01 vs001 kernel: [44532.883166] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2052. Nov 27 01:25:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2053. Nov 27 01:25:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2054. Nov 27 01:25:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2055. Nov 27 01:25:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2056. Nov 27 01:25:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2057. Nov 27 01:25:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2058. Nov 27 01:25:02 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2059. Nov 27 01:25:02 vs001 kernel: [44533.910597] block drbd16: State change failed: Device is held open by someone
  2060. Nov 27 01:25:02 vs001 kernel: [44533.912318] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2061. Nov 27 01:25:02 vs001 kernel: [44533.914034] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2062. Nov 27 01:25:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2063. Nov 27 01:25:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2064. Nov 27 01:25:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2065. Nov 27 01:25:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2066. Nov 27 01:25:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2067. Nov 27 01:25:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2068. Nov 27 01:25:03 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2069. Nov 27 01:25:03 vs001 kernel: [44534.941216] block drbd16: State change failed: Device is held open by someone
  2070. Nov 27 01:25:03 vs001 kernel: [44534.942931] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2071. Nov 27 01:25:03 vs001 kernel: [44534.944653] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2072. Nov 27 01:25:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2073. Nov 27 01:25:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2074. Nov 27 01:25:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2075. Nov 27 01:25:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2076. Nov 27 01:25:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2077. Nov 27 01:25:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2078. Nov 27 01:25:04 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2079. Nov 27 01:25:04 vs001 kernel: [44535.971800] block drbd16: State change failed: Device is held open by someone
  2080. Nov 27 01:25:04 vs001 kernel: [44535.973513] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2081. Nov 27 01:25:04 vs001 kernel: [44535.975230] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2082. Nov 27 01:25:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2083. Nov 27 01:25:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2084. Nov 27 01:25:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2085. Nov 27 01:25:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2086. Nov 27 01:25:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2087. Nov 27 01:25:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2088. Nov 27 01:25:05 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2089. Nov 27 01:25:05 vs001 kernel: [44537.002239] block drbd16: State change failed: Device is held open by someone
  2090. Nov 27 01:25:05 vs001 kernel: [44537.003961] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2091. Nov 27 01:25:05 vs001 kernel: [44537.005680] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2092. Nov 27 01:25:05 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2093. Nov 27 01:25:05 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2094. Nov 27 01:25:05 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2095. Nov 27 01:25:05 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2096. Nov 27 01:25:05 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2097. Nov 27 01:25:05 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2098. Nov 27 01:25:06 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2099. Nov 27 01:25:06 vs001 kernel: [44538.032885] block drbd16: State change failed: Device is held open by someone
  2100. Nov 27 01:25:06 vs001 kernel: [44538.034600] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2101. Nov 27 01:25:06 vs001 kernel: [44538.036332] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2102. Nov 27 01:25:06 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2103. Nov 27 01:25:06 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2104. Nov 27 01:25:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2105. Nov 27 01:25:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2106. Nov 27 01:25:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2107. Nov 27 01:25:06 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2108. Nov 27 01:25:07 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2109. Nov 27 01:25:07 vs001 kernel: [44539.063416] block drbd16: State change failed: Device is held open by someone
  2110. Nov 27 01:25:07 vs001 kernel: [44539.065131] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2111. Nov 27 01:25:07 vs001 kernel: [44539.066872] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2112. Nov 27 01:25:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2113. Nov 27 01:25:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2114. Nov 27 01:25:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2115. Nov 27 01:25:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2116. Nov 27 01:25:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2117. Nov 27 01:25:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2118. Nov 27 01:25:08 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2119. Nov 27 01:25:08 vs001 kernel: [44540.094043] block drbd16: State change failed: Device is held open by someone
  2120. Nov 27 01:25:08 vs001 kernel: [44540.095758] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2121. Nov 27 01:25:08 vs001 kernel: [44540.097480] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2122. Nov 27 01:25:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2123. Nov 27 01:25:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2124. Nov 27 01:25:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2125. Nov 27 01:25:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2126. Nov 27 01:25:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2127. Nov 27 01:25:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2128. Nov 27 01:25:09 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2129. Nov 27 01:25:09 vs001 kernel: [44541.124522] block drbd16: State change failed: Device is held open by someone
  2130. Nov 27 01:25:09 vs001 kernel: [44541.126235] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2131. Nov 27 01:25:09 vs001 kernel: [44541.127960] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2132. Nov 27 01:25:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2133. Nov 27 01:25:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2134. Nov 27 01:25:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2135. Nov 27 01:25:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2136. Nov 27 01:25:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2137. Nov 27 01:25:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2138. Nov 27 01:25:10 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2139. Nov 27 01:25:10 vs001 kernel: [44542.155435] block drbd16: State change failed: Device is held open by someone
  2140. Nov 27 01:25:10 vs001 kernel: [44542.157162] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2141. Nov 27 01:25:10 vs001 kernel: [44542.158881] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2142. Nov 27 01:25:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2143. Nov 27 01:25:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2144. Nov 27 01:25:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2145. Nov 27 01:25:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2146. Nov 27 01:25:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2147. Nov 27 01:25:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2148. Nov 27 01:25:11 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2149. Nov 27 01:25:11 vs001 kernel: [44543.186046] block drbd16: State change failed: Device is held open by someone
  2150. Nov 27 01:25:11 vs001 kernel: [44543.187765] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2151. Nov 27 01:25:11 vs001 kernel: [44543.189487] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2152. Nov 27 01:25:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2153. Nov 27 01:25:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2154. Nov 27 01:25:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2155. Nov 27 01:25:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2156. Nov 27 01:25:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2157. Nov 27 01:25:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2158. Nov 27 01:25:12 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2159. Nov 27 01:25:12 vs001 kernel: [44544.216570] block drbd16: State change failed: Device is held open by someone
  2160. Nov 27 01:25:12 vs001 kernel: [44544.218287] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2161. Nov 27 01:25:12 vs001 kernel: [44544.220017] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2162. Nov 27 01:25:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2163. Nov 27 01:25:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2164. Nov 27 01:25:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2165. Nov 27 01:25:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2166. Nov 27 01:25:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2167. Nov 27 01:25:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2168. Nov 27 01:25:13 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2169. Nov 27 01:25:13 vs001 kernel: [44545.247213] block drbd16: State change failed: Device is held open by someone
  2170. Nov 27 01:25:13 vs001 kernel: [44545.248938] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2171. Nov 27 01:25:13 vs001 kernel: [44545.250663] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2172. Nov 27 01:25:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2173. Nov 27 01:25:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2174. Nov 27 01:25:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2175. Nov 27 01:25:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2176. Nov 27 01:25:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2177. Nov 27 01:25:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2178. Nov 27 01:25:14 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2179. Nov 27 01:25:14 vs001 kernel: [44546.277995] block drbd16: State change failed: Device is held open by someone
  2180. Nov 27 01:25:14 vs001 kernel: [44546.279717] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2181. Nov 27 01:25:14 vs001 kernel: [44546.281439] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2182. Nov 27 01:25:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2183. Nov 27 01:25:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2184. Nov 27 01:25:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2185. Nov 27 01:25:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2186. Nov 27 01:25:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2187. Nov 27 01:25:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2188. Nov 27 01:25:15 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2189. Nov 27 01:25:15 vs001 kernel: [44547.308822] block drbd16: State change failed: Device is held open by someone
  2190. Nov 27 01:25:15 vs001 kernel: [44547.310541] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2191. Nov 27 01:25:15 vs001 kernel: [44547.312268] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2192. Nov 27 01:25:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2193. Nov 27 01:25:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2194. Nov 27 01:25:15 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2195. Nov 27 01:25:15 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2196. Nov 27 01:25:15 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2197. Nov 27 01:25:15 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2198. Nov 27 01:25:16 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2199. Nov 27 01:25:16 vs001 kernel: [44548.339731] block drbd16: State change failed: Device is held open by someone
  2200. Nov 27 01:25:16 vs001 kernel: [44548.341448] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2201. Nov 27 01:25:16 vs001 kernel: [44548.343172] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2202. Nov 27 01:25:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2203. Nov 27 01:25:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2204. Nov 27 01:25:16 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2205. Nov 27 01:25:16 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2206. Nov 27 01:25:16 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2207. Nov 27 01:25:16 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2208. Nov 27 01:25:17 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2209. Nov 27 01:25:17 vs001 kernel: [44549.364802] block drbd16: State change failed: Device is held open by someone
  2210. Nov 27 01:25:17 vs001 kernel: [44549.365968] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2211. Nov 27 01:25:17 vs001 kernel: [44549.366926] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2212. Nov 27 01:25:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2213. Nov 27 01:25:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2214. Nov 27 01:25:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2215. Nov 27 01:25:17 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2216. Nov 27 01:25:17 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2217. Nov 27 01:25:17 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2218. Nov 27 01:25:17 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2219. Nov 27 01:25:18 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2220. Nov 27 01:25:18 vs001 kernel: [44550.389195] block drbd16: State change failed: Device is held open by someone
  2221. Nov 27 01:25:18 vs001 kernel: [44550.390915] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2222. Nov 27 01:25:18 vs001 kernel: [44550.392646] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2223. Nov 27 01:25:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2224. Nov 27 01:25:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2225. Nov 27 01:25:18 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2226. Nov 27 01:25:18 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2227. Nov 27 01:25:18 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2228. Nov 27 01:25:18 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2229. Nov 27 01:25:19 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2230. Nov 27 01:25:19 vs001 kernel: [44551.414571] block drbd16: State change failed: Device is held open by someone
  2231. Nov 27 01:25:19 vs001 kernel: [44551.415471] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2232. Nov 27 01:25:19 vs001 kernel: [44551.416354] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2233. Nov 27 01:25:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2234. Nov 27 01:25:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2235. Nov 27 01:25:19 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2236. Nov 27 01:25:19 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2237. Nov 27 01:25:19 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2238. Nov 27 01:25:19 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2239. Nov 27 01:25:20 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2240. Nov 27 01:25:20 vs001 kernel: [44552.439460] block drbd16: State change failed: Device is held open by someone
  2241. Nov 27 01:25:20 vs001 kernel: [44552.441178] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2242. Nov 27 01:25:20 vs001 kernel: [44552.442899] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2243. Nov 27 01:25:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2244. Nov 27 01:25:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2245. Nov 27 01:25:20 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2246. Nov 27 01:25:20 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2247. Nov 27 01:25:20 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2248. Nov 27 01:25:20 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2249. Nov 27 01:25:21 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2250. Nov 27 01:25:21 vs001 kernel: [44553.469945] block drbd16: State change failed: Device is held open by someone
  2251. Nov 27 01:25:21 vs001 kernel: [44553.471664] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2252. Nov 27 01:25:21 vs001 kernel: [44553.473382] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2253. Nov 27 01:25:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2254. Nov 27 01:25:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2255. Nov 27 01:25:21 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2256. Nov 27 01:25:21 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2257. Nov 27 01:25:21 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2258. Nov 27 01:25:21 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2259. Nov 27 01:25:22 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2260. Nov 27 01:25:22 vs001 kernel: [44554.500388] block drbd16: State change failed: Device is held open by someone
  2261. Nov 27 01:25:22 vs001 kernel: [44554.502100] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2262. Nov 27 01:25:22 vs001 kernel: [44554.503828] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2263. Nov 27 01:25:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2264. Nov 27 01:25:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2265. Nov 27 01:25:22 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2266. Nov 27 01:25:22 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2267. Nov 27 01:25:22 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2268. Nov 27 01:25:22 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2269. Nov 27 01:25:23 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2270. Nov 27 01:25:23 vs001 kernel: [44555.531259] block drbd16: State change failed: Device is held open by someone
  2271. Nov 27 01:25:23 vs001 kernel: [44555.532972] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2272. Nov 27 01:25:23 vs001 kernel: [44555.534692] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2273. Nov 27 01:25:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2274. Nov 27 01:25:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2275. Nov 27 01:25:23 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2276. Nov 27 01:25:23 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2277. Nov 27 01:25:23 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2278. Nov 27 01:25:23 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2279. Nov 27 01:25:24 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2280. Nov 27 01:25:24 vs001 kernel: [44556.561860] block drbd16: State change failed: Device is held open by someone
  2281. Nov 27 01:25:24 vs001 kernel: [44556.563576] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2282. Nov 27 01:25:24 vs001 kernel: [44556.565295] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2283. Nov 27 01:25:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2284. Nov 27 01:25:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2285. Nov 27 01:25:24 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2286. Nov 27 01:25:24 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2287. Nov 27 01:25:24 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2288. Nov 27 01:25:24 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2289. Nov 27 01:25:25 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2290. Nov 27 01:25:25 vs001 kernel: [44557.592607] block drbd16: State change failed: Device is held open by someone
  2291. Nov 27 01:25:25 vs001 kernel: [44557.594318] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2292. Nov 27 01:25:25 vs001 kernel: [44557.596041] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2293. Nov 27 01:25:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2294. Nov 27 01:25:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2295. Nov 27 01:25:25 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2296. Nov 27 01:25:25 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2297. Nov 27 01:25:25 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2298. Nov 27 01:25:25 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2299. Nov 27 01:25:26 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2300. Nov 27 01:25:26 vs001 kernel: [44558.623149] block drbd16: State change failed: Device is held open by someone
  2301. Nov 27 01:25:26 vs001 kernel: [44558.624866] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2302. Nov 27 01:25:26 vs001 kernel: [44558.626584] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2303. Nov 27 01:25:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2304. Nov 27 01:25:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2305. Nov 27 01:25:26 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2306. Nov 27 01:25:26 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2307. Nov 27 01:25:26 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2308. Nov 27 01:25:26 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2309. Nov 27 01:25:27 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2310. Nov 27 01:25:27 vs001 kernel: [44559.653939] block drbd16: State change failed: Device is held open by someone
  2311. Nov 27 01:25:27 vs001 kernel: [44559.655659] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2312. Nov 27 01:25:27 vs001 kernel: [44559.657381] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2313. Nov 27 01:25:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2314. Nov 27 01:25:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2315. Nov 27 01:25:27 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2316. Nov 27 01:25:27 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2317. Nov 27 01:25:27 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2318. Nov 27 01:25:27 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2319. Nov 27 01:25:28 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2320. Nov 27 01:25:28 vs001 kernel: [44560.684450] block drbd16: State change failed: Device is held open by someone
  2321. Nov 27 01:25:28 vs001 kernel: [44560.686176] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2322. Nov 27 01:25:28 vs001 kernel: [44560.688121] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2323. Nov 27 01:25:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2324. Nov 27 01:25:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2325. Nov 27 01:25:28 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2326. Nov 27 01:25:28 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2327. Nov 27 01:25:28 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2328. Nov 27 01:25:28 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2329. Nov 27 01:25:29 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2330. Nov 27 01:25:29 vs001 kernel: [44561.714987] block drbd16: State change failed: Device is held open by someone
  2331. Nov 27 01:25:29 vs001 kernel: [44561.716702] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2332. Nov 27 01:25:29 vs001 kernel: [44561.718423] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2333. Nov 27 01:25:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2334. Nov 27 01:25:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2335. Nov 27 01:25:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2336. Nov 27 01:25:29 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2337. Nov 27 01:25:29 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2338. Nov 27 01:25:29 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2339. Nov 27 01:25:29 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2340. Nov 27 01:25:30 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2341. Nov 27 01:25:30 vs001 kernel: [44562.745721] block drbd16: State change failed: Device is held open by someone
  2342. Nov 27 01:25:30 vs001 kernel: [44562.747739] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2343. Nov 27 01:25:30 vs001 kernel: [44562.749721] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2344. Nov 27 01:25:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2345. Nov 27 01:25:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2346. Nov 27 01:25:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2347. Nov 27 01:25:30 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2348. Nov 27 01:25:30 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2349. Nov 27 01:25:30 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2350. Nov 27 01:25:30 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2351. Nov 27 01:25:31 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2352. Nov 27 01:25:32 vs001 kernel: [44563.777972] block drbd16: State change failed: Device is held open by someone
  2353. Nov 27 01:25:32 vs001 kernel: [44563.779702] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2354. Nov 27 01:25:32 vs001 kernel: [44563.781428] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2355. Nov 27 01:25:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2356. Nov 27 01:25:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2357. Nov 27 01:25:32 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2358. Nov 27 01:25:32 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2359. Nov 27 01:25:32 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2360. Nov 27 01:25:32 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2361. Nov 27 01:25:33 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2362. Nov 27 01:25:33 vs001 kernel: [44564.809049] block drbd16: State change failed: Device is held open by someone
  2363. Nov 27 01:25:33 vs001 kernel: [44564.810775] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2364. Nov 27 01:25:33 vs001 kernel: [44564.812502] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2365. Nov 27 01:25:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2366. Nov 27 01:25:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2367. Nov 27 01:25:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2368. Nov 27 01:25:33 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2369. Nov 27 01:25:33 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2370. Nov 27 01:25:33 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2371. Nov 27 01:25:33 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2372. Nov 27 01:25:34 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2373. Nov 27 01:25:34 vs001 kernel: [44565.839887] block drbd16: State change failed: Device is held open by someone
  2374. Nov 27 01:25:34 vs001 kernel: [44565.841609] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2375. Nov 27 01:25:34 vs001 kernel: [44565.843339] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2376. Nov 27 01:25:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2377. Nov 27 01:25:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2378. Nov 27 01:25:34 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2379. Nov 27 01:25:34 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2380. Nov 27 01:25:34 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2381. Nov 27 01:25:34 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2382. Nov 27 01:25:35 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2383. Nov 27 01:25:35 vs001 kernel: [44566.870272] block drbd16: State change failed: Device is held open by someone
  2384. Nov 27 01:25:35 vs001 kernel: [44566.872004] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2385. Nov 27 01:25:35 vs001 kernel: [44566.873730] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2386. Nov 27 01:25:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2387. Nov 27 01:25:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2388. Nov 27 01:25:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2389. Nov 27 01:25:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2390. Nov 27 01:25:35 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2391. Nov 27 01:25:35 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2392. Nov 27 01:25:36 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2393. Nov 27 01:25:36 vs001 kernel: [44567.900938] block drbd16: State change failed: Device is held open by someone
  2394. Nov 27 01:25:36 vs001 kernel: [44567.902663] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2395. Nov 27 01:25:36 vs001 kernel: [44567.904408] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2396. Nov 27 01:25:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2397. Nov 27 01:25:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2398. Nov 27 01:25:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2399. Nov 27 01:25:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2400. Nov 27 01:25:36 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2401. Nov 27 01:25:36 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2402. Nov 27 01:25:37 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2403. Nov 27 01:25:37 vs001 kernel: [44568.931625] block drbd16: State change failed: Device is held open by someone
  2404. Nov 27 01:25:37 vs001 kernel: [44568.933349] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2405. Nov 27 01:25:37 vs001 kernel: [44568.935094] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2406. Nov 27 01:25:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2407. Nov 27 01:25:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2408. Nov 27 01:25:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2409. Nov 27 01:25:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2410. Nov 27 01:25:37 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2411. Nov 27 01:25:37 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2412. Nov 27 01:25:38 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2413. Nov 27 01:25:38 vs001 kernel: [44569.962381] block drbd16: State change failed: Device is held open by someone
  2414. Nov 27 01:25:38 vs001 kernel: [44569.964115] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2415. Nov 27 01:25:38 vs001 kernel: [44569.965845] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2416. Nov 27 01:25:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2417. Nov 27 01:25:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2418. Nov 27 01:25:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2419. Nov 27 01:25:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2420. Nov 27 01:25:38 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2421. Nov 27 01:25:38 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2422. Nov 27 01:25:39 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2423. Nov 27 01:25:39 vs001 kernel: [44570.993146] block drbd16: State change failed: Device is held open by someone
  2424. Nov 27 01:25:39 vs001 kernel: [44570.994873] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2425. Nov 27 01:25:39 vs001 kernel: [44570.996598] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2426. Nov 27 01:25:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2427. Nov 27 01:25:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2428. Nov 27 01:25:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2429. Nov 27 01:25:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2430. Nov 27 01:25:39 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2431. Nov 27 01:25:39 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2432. Nov 27 01:25:40 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2433. Nov 27 01:25:40 vs001 kernel: [44572.024021] block drbd16: State change failed: Device is held open by someone
  2434. Nov 27 01:25:40 vs001 kernel: [44572.025747] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2435. Nov 27 01:25:40 vs001 kernel: [44572.027481] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2436. Nov 27 01:25:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2437. Nov 27 01:25:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2438. Nov 27 01:25:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2439. Nov 27 01:25:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2440. Nov 27 01:25:40 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2441. Nov 27 01:25:40 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2442. Nov 27 01:25:41 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2443. Nov 27 01:25:41 vs001 kernel: [44573.054784] block drbd16: State change failed: Device is held open by someone
  2444. Nov 27 01:25:41 vs001 kernel: [44573.056510] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2445. Nov 27 01:25:41 vs001 kernel: [44573.058243] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2446. Nov 27 01:25:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2447. Nov 27 01:25:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2448. Nov 27 01:25:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2449. Nov 27 01:25:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2450. Nov 27 01:25:41 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2451. Nov 27 01:25:41 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2452. Nov 27 01:25:42 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2453. Nov 27 01:25:42 vs001 kernel: [44574.085769] block drbd16: State change failed: Device is held open by someone
  2454. Nov 27 01:25:42 vs001 kernel: [44574.087504] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2455. Nov 27 01:25:42 vs001 kernel: [44574.089234] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2456. Nov 27 01:25:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2457. Nov 27 01:25:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2458. Nov 27 01:25:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2459. Nov 27 01:25:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2460. Nov 27 01:25:42 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2461. Nov 27 01:25:42 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2462. Nov 27 01:25:43 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2463. Nov 27 01:25:43 vs001 kernel: [44575.116561] block drbd16: State change failed: Device is held open by someone
  2464. Nov 27 01:25:43 vs001 kernel: [44575.118284] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2465. Nov 27 01:25:43 vs001 kernel: [44575.120021] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2466. Nov 27 01:25:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2467. Nov 27 01:25:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2468. Nov 27 01:25:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2469. Nov 27 01:25:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2470. Nov 27 01:25:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2471. Nov 27 01:25:43 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2472. Nov 27 01:25:43 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2473. Nov 27 01:25:44 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2474. Nov 27 01:25:44 vs001 kernel: [44576.147537] block drbd16: State change failed: Device is held open by someone
  2475. Nov 27 01:25:44 vs001 kernel: [44576.149259] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2476. Nov 27 01:25:44 vs001 kernel: [44576.150993] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2477. Nov 27 01:25:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2478. Nov 27 01:25:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2479. Nov 27 01:25:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2480. Nov 27 01:25:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2481. Nov 27 01:25:44 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2482. Nov 27 01:25:44 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2483. Nov 27 01:25:45 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2484. Nov 27 01:25:45 vs001 kernel: [44577.177900] block drbd16: State change failed: Device is held open by someone
  2485. Nov 27 01:25:45 vs001 kernel: [44577.179632] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2486. Nov 27 01:25:45 vs001 kernel: [44577.181362] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2487. Nov 27 01:25:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2488. Nov 27 01:25:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2489. Nov 27 01:25:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2490. Nov 27 01:25:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2491. Nov 27 01:25:45 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2492. Nov 27 01:25:45 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2493. Nov 27 01:25:46 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2494. Nov 27 01:25:46 vs001 kernel: [44578.208608] block drbd16: State change failed: Device is held open by someone
  2495. Nov 27 01:25:46 vs001 kernel: [44578.210328] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2496. Nov 27 01:25:46 vs001 kernel: [44578.212058] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2497. Nov 27 01:25:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2498. Nov 27 01:25:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2499. Nov 27 01:25:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2500. Nov 27 01:25:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2501. Nov 27 01:25:46 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2502. Nov 27 01:25:46 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2503. Nov 27 01:25:47 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2504. Nov 27 01:25:47 vs001 kernel: [44579.238914] block drbd16: State change failed: Device is held open by someone
  2505. Nov 27 01:25:47 vs001 kernel: [44579.240628] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2506. Nov 27 01:25:47 vs001 kernel: [44579.242345] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2507. Nov 27 01:25:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2508. Nov 27 01:25:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2509. Nov 27 01:25:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2510. Nov 27 01:25:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2511. Nov 27 01:25:47 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2512. Nov 27 01:25:47 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2513. Nov 27 01:25:48 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2514. Nov 27 01:25:48 vs001 kernel: [44580.268019] block drbd16: State change failed: Device is held open by someone
  2515. Nov 27 01:25:48 vs001 kernel: [44580.269749] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2516. Nov 27 01:25:48 vs001 kernel: [44580.271473] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2517. Nov 27 01:25:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2518. Nov 27 01:25:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2519. Nov 27 01:25:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2520. Nov 27 01:25:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2521. Nov 27 01:25:48 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2522. Nov 27 01:25:48 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2523. Nov 27 01:25:49 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2524. Nov 27 01:25:49 vs001 kernel: [44581.298727] block drbd16: State change failed: Device is held open by someone
  2525. Nov 27 01:25:49 vs001 kernel: [44581.300437] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2526. Nov 27 01:25:49 vs001 kernel: [44581.302151] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2527. Nov 27 01:25:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2528. Nov 27 01:25:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2529. Nov 27 01:25:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2530. Nov 27 01:25:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2531. Nov 27 01:25:49 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2532. Nov 27 01:25:49 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2533. Nov 27 01:25:50 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2534. Nov 27 01:25:50 vs001 kernel: [44582.329523] block drbd16: State change failed: Device is held open by someone
  2535. Nov 27 01:25:50 vs001 kernel: [44582.331240] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2536. Nov 27 01:25:50 vs001 kernel: [44582.332954] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2537. Nov 27 01:25:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2538. Nov 27 01:25:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2539. Nov 27 01:25:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2540. Nov 27 01:25:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2541. Nov 27 01:25:50 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2542. Nov 27 01:25:50 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2543. Nov 27 01:25:51 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2544. Nov 27 01:25:51 vs001 kernel: [44583.359563] block drbd16: State change failed: Device is held open by someone
  2545. Nov 27 01:25:51 vs001 kernel: [44583.360722] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2546. Nov 27 01:25:51 vs001 kernel: [44583.361890] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2547. Nov 27 01:25:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2548. Nov 27 01:25:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2549. Nov 27 01:25:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2550. Nov 27 01:25:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2551. Nov 27 01:25:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2552. Nov 27 01:25:51 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2553. Nov 27 01:25:51 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2554. Nov 27 01:25:52 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2555. Nov 27 01:25:52 vs001 kernel: [44584.384596] block drbd16: State change failed: Device is held open by someone
  2556. Nov 27 01:25:52 vs001 kernel: [44584.386303] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2557. Nov 27 01:25:52 vs001 kernel: [44584.388017] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2558. Nov 27 01:25:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2559. Nov 27 01:25:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2560. Nov 27 01:25:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2561. Nov 27 01:25:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2562. Nov 27 01:25:52 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2563. Nov 27 01:25:52 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2564. Nov 27 01:25:53 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2565. Nov 27 01:25:53 vs001 kernel: [44585.412634] block drbd16: State change failed: Device is held open by someone
  2566. Nov 27 01:25:53 vs001 kernel: [44585.413501] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2567. Nov 27 01:25:53 vs001 kernel: [44585.414374] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2568. Nov 27 01:25:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2569. Nov 27 01:25:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2570. Nov 27 01:25:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2571. Nov 27 01:25:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2572. Nov 27 01:25:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2573. Nov 27 01:25:53 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2574. Nov 27 01:25:53 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2575. Nov 27 01:25:54 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2576. Nov 27 01:25:54 vs001 kernel: [44586.433622] block drbd16: State change failed: Device is held open by someone
  2577. Nov 27 01:25:54 vs001 kernel: [44586.435338] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2578. Nov 27 01:25:54 vs001 kernel: [44586.437052] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2579. Nov 27 01:25:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2580. Nov 27 01:25:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2581. Nov 27 01:25:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2582. Nov 27 01:25:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2583. Nov 27 01:25:54 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2584. Nov 27 01:25:54 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2585. Nov 27 01:25:55 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2586. Nov 27 01:25:55 vs001 kernel: [44587.464240] block drbd16: State change failed: Device is held open by someone
  2587. Nov 27 01:25:55 vs001 kernel: [44587.465947] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2588. Nov 27 01:25:55 vs001 kernel: [44587.467670] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2589. Nov 27 01:25:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2590. Nov 27 01:25:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2591. Nov 27 01:25:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2592. Nov 27 01:25:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2593. Nov 27 01:25:55 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2594. Nov 27 01:25:55 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2595. Nov 27 01:25:56 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2596. Nov 27 01:25:56 vs001 kernel: [44588.494561] block drbd16: State change failed: Device is held open by someone
  2597. Nov 27 01:25:56 vs001 kernel: [44588.496404] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2598. Nov 27 01:25:56 vs001 kernel: [44588.498332] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2599. Nov 27 01:25:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2600. Nov 27 01:25:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2601. Nov 27 01:25:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2602. Nov 27 01:25:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2603. Nov 27 01:25:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2604. Nov 27 01:25:56 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2605. Nov 27 01:25:56 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2606. Nov 27 01:25:57 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2607. Nov 27 01:25:57 vs001 kernel: [44589.525155] block drbd16: State change failed: Device is held open by someone
  2608. Nov 27 01:25:57 vs001 kernel: [44589.526886] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2609. Nov 27 01:25:57 vs001 kernel: [44589.528603] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2610. Nov 27 01:25:57 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2611. Nov 27 01:25:57 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2612. Nov 27 01:25:57 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2613. Nov 27 01:25:57 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2614. Nov 27 01:25:57 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2615. Nov 27 01:25:57 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2616. Nov 27 01:25:58 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2617. Nov 27 01:25:58 vs001 kernel: [44590.555786] block drbd16: State change failed: Device is held open by someone
  2618. Nov 27 01:25:58 vs001 kernel: [44590.557494] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2619. Nov 27 01:25:58 vs001 kernel: [44590.559217] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2620. Nov 27 01:25:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2621. Nov 27 01:25:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2622. Nov 27 01:25:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2623. Nov 27 01:25:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2624. Nov 27 01:25:58 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2625. Nov 27 01:25:58 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2626. Nov 27 01:25:59 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2627. Nov 27 01:25:59 vs001 kernel: [44591.586574] block drbd16: State change failed: Device is held open by someone
  2628. Nov 27 01:25:59 vs001 kernel: [44591.588283] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2629. Nov 27 01:25:59 vs001 kernel: [44591.589998] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2630. Nov 27 01:25:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2631. Nov 27 01:25:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2632. Nov 27 01:25:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2633. Nov 27 01:25:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2634. Nov 27 01:25:59 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2635. Nov 27 01:25:59 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2636. Nov 27 01:26:00 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2637. Nov 27 01:26:00 vs001 kernel: [44592.617203] block drbd16: State change failed: Device is held open by someone
  2638. Nov 27 01:26:00 vs001 kernel: [44592.618921] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2639. Nov 27 01:26:00 vs001 kernel: [44592.620639] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2640. Nov 27 01:26:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2641. Nov 27 01:26:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2642. Nov 27 01:26:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2643. Nov 27 01:26:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2644. Nov 27 01:26:00 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2645. Nov 27 01:26:00 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2646. Nov 27 01:26:01 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2647. Nov 27 01:26:01 vs001 kernel: [44593.647818] block drbd16: State change failed: Device is held open by someone
  2648. Nov 27 01:26:01 vs001 kernel: [44593.649529] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2649. Nov 27 01:26:01 vs001 kernel: [44593.651253] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2650. Nov 27 01:26:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2651. Nov 27 01:26:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2652. Nov 27 01:26:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2653. Nov 27 01:26:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2654. Nov 27 01:26:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2655. Nov 27 01:26:01 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2656. Nov 27 01:26:01 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2657. Nov 27 01:26:02 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2658. Nov 27 01:26:02 vs001 kernel: [44594.678709] block drbd16: State change failed: Device is held open by someone
  2659. Nov 27 01:26:02 vs001 kernel: [44594.680417] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2660. Nov 27 01:26:02 vs001 kernel: [44594.682140] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2661. Nov 27 01:26:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2662. Nov 27 01:26:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2663. Nov 27 01:26:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2664. Nov 27 01:26:02 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2665. Nov 27 01:26:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2666. Nov 27 01:26:02 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2667. Nov 27 01:26:03 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2668. Nov 27 01:26:03 vs001 kernel: [44595.709114] block drbd16: State change failed: Device is held open by someone
  2669. Nov 27 01:26:03 vs001 kernel: [44595.710840] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2670. Nov 27 01:26:03 vs001 kernel: [44595.712560] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2671. Nov 27 01:26:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2672. Nov 27 01:26:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2673. Nov 27 01:26:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2674. Nov 27 01:26:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2675. Nov 27 01:26:03 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2676. Nov 27 01:26:03 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2677. Nov 27 01:26:04 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2678. Nov 27 01:26:04 vs001 kernel: [44596.739896] block drbd16: State change failed: Device is held open by someone
  2679. Nov 27 01:26:04 vs001 kernel: [44596.741613] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2680. Nov 27 01:26:04 vs001 kernel: [44596.743342] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2681. Nov 27 01:26:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2682. Nov 27 01:26:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2683. Nov 27 01:26:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2684. Nov 27 01:26:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2685. Nov 27 01:26:04 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2686. Nov 27 01:26:04 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2687. Nov 27 01:26:05 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2688. Nov 27 01:26:05 vs001 kernel: [44597.770846] block drbd16: State change failed: Device is held open by someone
  2689. Nov 27 01:26:05 vs001 kernel: [44597.772566] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2690. Nov 27 01:26:05 vs001 kernel: [44597.774312] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2691. Nov 27 01:26:05 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2692. Nov 27 01:26:05 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2693. Nov 27 01:26:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2694. Nov 27 01:26:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2695. Nov 27 01:26:06 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2696. Nov 27 01:26:06 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2697. Nov 27 01:26:07 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2698. Nov 27 01:26:07 vs001 kernel: [44598.801603] block drbd16: State change failed: Device is held open by someone
  2699. Nov 27 01:26:07 vs001 kernel: [44598.803331] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2700. Nov 27 01:26:07 vs001 kernel: [44598.805057] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2701. Nov 27 01:26:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2702. Nov 27 01:26:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2703. Nov 27 01:26:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2704. Nov 27 01:26:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2705. Nov 27 01:26:07 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2706. Nov 27 01:26:07 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2707. Nov 27 01:26:08 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2708. Nov 27 01:26:08 vs001 kernel: [44599.832193] block drbd16: State change failed: Device is held open by someone
  2709. Nov 27 01:26:08 vs001 kernel: [44599.833929] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2710. Nov 27 01:26:08 vs001 kernel: [44599.835655] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2711. Nov 27 01:26:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2712. Nov 27 01:26:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2713. Nov 27 01:26:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2714. Nov 27 01:26:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2715. Nov 27 01:26:08 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2716. Nov 27 01:26:08 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2717. Nov 27 01:26:09 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2718. Nov 27 01:26:09 vs001 kernel: [44600.862901] block drbd16: State change failed: Device is held open by someone
  2719. Nov 27 01:26:09 vs001 kernel: [44600.864621] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2720. Nov 27 01:26:09 vs001 kernel: [44600.866353] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2721. Nov 27 01:26:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2722. Nov 27 01:26:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2723. Nov 27 01:26:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2724. Nov 27 01:26:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2725. Nov 27 01:26:09 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2726. Nov 27 01:26:09 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2727. Nov 27 01:26:10 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2728. Nov 27 01:26:10 vs001 kernel: [44601.893583] block drbd16: State change failed: Device is held open by someone
  2729. Nov 27 01:26:10 vs001 kernel: [44601.895307] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2730. Nov 27 01:26:10 vs001 kernel: [44601.897029] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2731. Nov 27 01:26:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2732. Nov 27 01:26:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2733. Nov 27 01:26:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2734. Nov 27 01:26:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2735. Nov 27 01:26:10 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2736. Nov 27 01:26:10 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2737. Nov 27 01:26:11 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2738. Nov 27 01:26:11 vs001 kernel: [44602.924323] block drbd16: State change failed: Device is held open by someone
  2739. Nov 27 01:26:11 vs001 kernel: [44602.926048] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2740. Nov 27 01:26:11 vs001 kernel: [44602.927773] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2741. Nov 27 01:26:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2742. Nov 27 01:26:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary
  2743. Nov 27 01:26:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2744. Nov 27 01:26:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2745. Nov 27 01:26:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2746. Nov 27 01:26:11 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2747. Nov 27 01:26:11 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2748. Nov 27 01:26:12 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2749. Nov 27 01:26:12 vs001 kernel: [44603.955059] block drbd16: State change failed: Device is held open by someone
  2750. Nov 27 01:26:12 vs001 kernel: [44603.956775] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2751. Nov 27 01:26:12 vs001 kernel: [44603.958514] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2752. Nov 27 01:26:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone#012Command 'drbdsetup 16 secondary
  2753. Nov 27 01:26:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2754. Nov 27 01:26:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2755. Nov 27 01:26:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2756. Nov 27 01:26:12 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2757. Nov 27 01:26:12 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2758. Nov 27 01:26:13 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2759. Nov 27 01:26:13 vs001 kernel: [44604.985902] block drbd16: State change failed: Device is held open by someone
  2760. Nov 27 01:26:13 vs001 kernel: [44604.987620] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2761. Nov 27 01:26:13 vs001 kernel: [44604.989345] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2762. Nov 27 01:26:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone#012Command 'drbdsetup 16 secondary
  2763. Nov 27 01:26:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2764. Nov 27 01:26:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2765. Nov 27 01:26:13 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2766. Nov 27 01:26:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2767. Nov 27 01:26:13 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) ' terminated with exit code 11
  2768. Nov 27 01:26:14 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2769. Nov 27 01:26:14 vs001 kernel: [44606.016353] block drbd16: State change failed: Device is held open by someone
  2770. Nov 27 01:26:14 vs001 kernel: [44606.018078] block drbd16: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r--- }
  2771. Nov 27 01:26:14 vs001 kernel: [44606.019802] block drbd16: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r--- }
  2772. Nov 27 01:26:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) 16: State change failed: (-12) Device is held open by someone
  2773. Nov 27 01:26:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stderr) Command 'drbdsetup 16 secondary' terminated with exit code 11
  2774. Nov 27 01:26:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Called drbdadm -c /etc/drbd.conf secondary cinder-HDD1
  2775. Nov 27 01:26:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Exit code 11
  2776. Nov 27 01:26:14 vs001 drbd[7999]: ERROR: cinder-HDD1: Command output:
  2777. Nov 27 01:26:14 vs001 lrmd: [1013]: info: RA output: (drbd_cinder-HDD1:1:stop:stdout)
  2778. Nov 27 01:26:15 vs001 drbd[7999]: WARNING: cinder-HDD1 still Primary, demoting.
  2779. Nov 27 01:26:15 vs001 lrmd: [1013]: WARN: drbd_cinder-HDD1:1:stop process (PID 7999) timed out (try 1). Killing with signal SIGTERM (15).
  2780. Nov 27 01:26:15 vs001 lrmd: [1013]: WARN: operation stop[316] on drbd_cinder-HDD1:1 for client 1016: pid 7999 timed out
  2781. Nov 27 01:26:15 vs001 crmd: [1016]: ERROR: process_lrm_event: LRM operation drbd_cinder-HDD1:1_stop_0 (316) Timed Out (timeout=100000ms)
  2782. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  2783. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_cinder-HDD1:1 (INFINITY)
  2784. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 626: fail-count-drbd_cinder-HDD1:1=INFINITY
  2785. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  2786. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_cinder-HDD1:1 (1385511975)
  2787. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 628: last-failure-drbd_cinder-HDD1:1=1385511975
  2788. Nov 27 01:26:15 vs001 crmd: [1016]: info: do_lrm_rsc_op: Performing key=571:4864:0:5b2330eb-4efe-4171-8267-f8e5c0ad4452 op=drbd_cinder-HDD1:1_monitor_31000 )
  2789. Nov 27 01:26:15 vs001 lrmd: [1013]: info: rsc:drbd_cinder-HDD1:1 monitor[317] (pid 11424)
  2790. Nov 27 01:26:15 vs001 lrmd: [1013]: info: operation monitor[317] on drbd_cinder-HDD1:1 for client 1016: pid 11424 exited with return code 8
  2791. Nov 27 01:26:15 vs001 crmd: [1016]: info: process_lrm_event: LRM operation drbd_cinder-HDD1:1_monitor_31000 (call=317, rc=8, cib-update=235, confirmed=false) master
  2792. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
  2793. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_cinder-HDD1:1 (1000000)
  2794. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_perform_update: Sent update 630: fail-count-drbd_cinder-HDD1:1=1000000
  2795. Nov 27 01:26:15 vs001 attrd: [1014]: notice: attrd_ais_dispatch: Update relayed from vs003
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!