API tools faq
paste
digimer

Untitled

digimer
Jul 14th, 2020
155
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 114.85 KB | None | 0 0
raw download clone embed print report
  1. # Node 2 logs below
  2. ====
  3. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was member)
  4. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (handle_shutdown_request) info: Creating shutdown request for el8-a01n02 (state=S_IDLE)
  5. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594755459 | from el8-a01n01
  6. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.44 2
  7. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.45 (null)
  8. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=45
  9. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/>
  10. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ <instance_attributes id="status-2">
  11. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ <nvpair id="status-2-shutdown" name="shutdown" value="1594755459"/>
  12. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ </instance_attributes>
  13. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ </transient_attributes>
  14. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.16.45)
  15. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 0 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.16.45 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=true
  16. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
  17. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status_fencing) info: Node el8-a01n01 is active
  18. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is online
  19. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n02 is shutting down
  20. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  21. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  22. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  23. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  24. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  25. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02
  26. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  27. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  28. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  29. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  30. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  31. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  32. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  33. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n02
  34. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Shutdown el8-a01n02
  35. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node1 (Started el8-a01n01)
  36. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Move virsh_node2 ( el8-a01n02 -> el8-a01n01 )
  37. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  38. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop hypervisor:1 ( el8-a01n02 ) due to node availability
  39. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  40. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  41. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-43.bz2
  42. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  43. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_te_invoke) info: Processing graph 1 (ref=pe_calc-dc-1594755459-31) derived from /var/lib/pacemaker/pengine/pe-input-43.bz2
  44. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n02 | action 9
  45. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 on el8-a01n02 | action 14
  46. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 21
  47. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.45 2
  48. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.46 (null)
  49. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=46
  50. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0, @
  51. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.16.46)
  52. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.46 2
  53. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.47 (null)
  54. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=47
  55. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0
  56. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.16.47)
  57. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.47 2
  58. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.48 (null)
  59. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=48
  60. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=27, @rc-code=0, @op-status=0, @exec-time=1
  61. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 1 action 9 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=27
  62. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.16.48)
  63. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.48 2
  64. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.49 (null)
  65. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=49
  66. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0
  67. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n01 | action 10
  68. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_lrm_rsc_op) info: Performing key=10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=virsh_node2_start_0
  69. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_execute) info: executing - rsc:virsh_node2 action:start call_id:26
  70. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.16.49)
  71. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52)
  72. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.49 2
  73. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.50 (null)
  74. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=50
  75. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459
  76. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/52, version=0.16.50)
  77. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemakerd [12240] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  78. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemakerd [12240] (pcmk_shutdown_worker) notice: Shutting down Pacemaker
  79. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemakerd [12240] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 12246
  80. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  81. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms
  82. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_log) warning: Input I_SHUTDOWN received in state S_TRANSITION_ENGINE from crm_shutdown
  83. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown
  84. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n01)
  85. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member)
  86. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_POLICY_ENGINE)
  87. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594755459 | from el8-a01n01
  88. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.50 2
  89. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.51 (null)
  90. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=51
  91. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
  92. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ <instance_attributes id="status-1">
  93. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ <nvpair id="status-1-shutdown" name="shutdown" value="1594755459"/>
  94. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ </instance_attributes>
  95. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ </transient_attributes>
  96. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.16.51)
  97. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) notice: Transition 1 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.16.51 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=false
  98. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_finished) info: finished - rsc:virsh_node2 action:start call_id:26 exit-code:0 exec-time:704ms queue-time:0ms
  99. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) notice: Result of start operation for virsh_node2 on el8-a01n01: 0 (ok) | call=26 key=virsh_node2_start_0 confirmed=true cib-update=53
  100. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/53)
  101. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.51 2
  102. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.52 (null)
  103. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=52
  104. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=26, @rc-code=0, @op-status=0, @exec-time=704
  105. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/53, version=0.16.52)
  106. Jul 14 19:37:39 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 1 action 10 (virsh_node2_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=26
  107. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.52 2
  108. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.53 (null)
  109. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=53
  110. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2253
  111. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 1 action 14 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=29
  112. Jul 14 19:37:41 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.16.53)
  113. Jul 14 19:37:46 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 12bcdc96a01db61039f2acac78431a32 for 0.16.53 (0x55f7b2ea4e40 0)
  114.  
  115. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.53 2
  116. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.54 (null)
  117. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=54
  118. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=2:198;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=31, @rc-code=198, @op-status=2, @exec-time=99989
  119. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_last_failure_0" operation_key="drbd_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629" transition-magic="2:198;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629" exit-reason="" on_node="el8-a01n02" call-id="31" rc-code="
  120. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.16.54)
  121. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 1 aborted by operation drbd_stop_0 'modify' on el8-a01n02: Event failed | magic=2:198;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629 cib=0.16.54 source=process_graph_event:453 complete=false
  122. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (update_failcount) info: Updating failcount for drbd on el8-a01n02 after failed stop: rc=198 (update=INFINITY, time=1594755559)
  123. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) notice: Transition 1 action 21 (drbd_stop_0 on el8-a01n02): expected 'ok' but got 'OCF_TIMEOUT' | target-rc=0 rc=198 call-id=31
  124. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
  125. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (run_graph) notice: Transition 1 (Complete=8, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-43.bz2): Stopped
  126. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594755559 | from el8-a01n01
  127. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.54 2
  128. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.55 (null)
  129. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=55
  130. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-fail-count-drbd.stop_0" name="fail-count-drbd#stop_0" value="INFINITY"/>
  131. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.16.55)
  132. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.55 2
  133. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.56 (null)
  134. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=56
  135. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-drbd.stop_0" name="last-failure-drbd#stop_0" value="1594755559"/>
  136. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/8, version=0.16.56)
  137. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 1 aborted by status-2-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change | cib=0.16.55 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  138. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 1 aborted by status-2-last-failure-drbd.stop_0 doing create last-failure-drbd#stop_0=1594755559: Transient attribute change | cib=0.16.56 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  139. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is shutting down
  140. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n02 is shutting down
  141. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 19:37:39 2020 | rc=198 id=drbd_last_failure_0
  142. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 19:37:39 2020 | rc=198 id=drbd_last_0
  143. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  144. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  145. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  146. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  147. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  148. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  149. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n01
  150. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  151. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 ]
  152. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n02 ]
  153. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  154. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  155. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 ]
  156. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node1 cannot run anywhere
  157. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node2 cannot run anywhere
  158. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:0 cannot run anywhere
  159. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  160. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:0 cannot run anywhere
  161. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  162. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  163. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  164. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  165. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Shutdown el8-a01n01
  166. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  167. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop virsh_node1 ( el8-a01n01 ) due to node availability
  168. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop virsh_node2 ( el8-a01n01 ) due to node availability
  169. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability
  170. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:1 (Stopped)
  171. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  172. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  173. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) warning: Calculated transition 2 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-7.bz2
  174. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (handle_response) info: pe_calc calculation pe_calc-dc-1594755559-37 is obsolete
  175. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is shutting down
  176. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n02 is shutting down
  177. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 19:37:39 2020 | rc=198 id=drbd_last_failure_0
  178. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 19:37:39 2020 | rc=198 id=drbd_last_0
  179. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  180. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  181. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  182. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  183. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  184. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  185. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n01
  186. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  187. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 ]
  188. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n02 ]
  189. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  190. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  191. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Started: [ el8-a01n01 ]
  192. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  193. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  194. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  195. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  196. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node1 cannot run anywhere
  197. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node2 cannot run anywhere
  198. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:0 cannot run anywhere
  199. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  200. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:0 cannot run anywhere
  201. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  202. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  203. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  204. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  205. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Shutdown el8-a01n01
  206. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  207. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop virsh_node1 ( el8-a01n01 ) due to node availability
  208. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop virsh_node2 ( el8-a01n01 ) due to node availability
  209. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability
  210. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:1 (Stopped)
  211. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  212. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  213. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) warning: Calculated transition 3 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-8.bz2
  214. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  215. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_te_invoke) info: Processing graph 3 (ref=pe_calc-dc-1594755559-38) derived from /var/lib/pacemaker/pengine/pe-warn-8.bz2
  216. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 locally on el8-a01n01 | action 6
  217. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_lrm_rsc_op) info: Performing key=6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=virsh_node1_stop_0
  218. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57)
  219. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_execute) info: executing - rsc:virsh_node1 action:stop call_id:28
  220. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.56 2
  221. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.57 (null)
  222. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=57
  223. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  224. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_finished) info: finished - rsc:virsh_node1 action:stop call_id:28 exit-code:0 exec-time:0ms queue-time:0ms
  225. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.16.57)
  226. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n01 | action 7
  227. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_lrm_rsc_op) info: Performing key=7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=virsh_node2_stop_0
  228. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:29
  229. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:29 exit-code:0 exec-time:1ms queue-time:0ms
  230. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58)
  231. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.57 2
  232. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.58 (null)
  233. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=58
  234. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  235. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.16.58)
  236. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=5 timeout=60000
  237. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (handle_request) notice: Client pacemaker-controld.12246.215b449b wants to fence (reboot) 'el8-a01n02' with device '(any)'
  238. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=7224bb7a-7dc4-44c1-a9ce-0e5491b6e52a state=0
  239. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  240. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  241. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 7224bb7a-7dc4-44c1-a9ce-0e5491b6e52a
  242. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 7224bb7a-7dc4-44c1-a9ce-0e5491b6e52a
  243. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  244. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.12246|id=7224bb7a-7dc4-44c1-a9ce-0e5491b6e52a
  245. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.12246 (72s, 0s)
  246. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  247. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  248. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-fenced [12242] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  249. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) info: Result of monitor operation for virsh_node1 on el8-a01n01: Cancelled | call=22 key=virsh_node1_monitor_60000 confirmed=true
  250. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok) | call=28 key=virsh_node1_stop_0 confirmed=true cib-update=59
  251. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/59)
  252. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.58 2
  253. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.59 (null)
  254. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=59
  255. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=28, @rc-code=0, @op-status=0
  256. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/59, version=0.16.59)
  257. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/60)
  258. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.59 2
  259. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.60 (null)
  260. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=60
  261. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=29, @rc-code=0, @op-status=0, @exec-time=1
  262. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/60, version=0.16.60)
  263. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n01: 0 (ok) | call=29 key=virsh_node2_stop_0 confirmed=true cib-update=60
  264. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 3 action 6 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=28
  265. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 3 action 7 (virsh_node2_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=29
  266. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n01 | action 8
  267. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000
  268. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_lrm_rsc_op) info: Performing key=8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=hypervisor_stop_0
  269. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_execute) info: executing - rsc:hypervisor action:stop call_id:31
  270. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61)
  271. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.60 2
  272. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.61 (null)
  273. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=61
  274. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  275. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.16.61)
  276. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n01: Cancelled | call=24 key=hypervisor_monitor_60000 confirmed=true
  277. Jul 14 19:39:19 el8-a01n01.digimer.ca pacemaker-execd [12243] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1175
  278. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok) | call=31 key=hypervisor_stop_0 confirmed=true cib-update=62
  279. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/62)
  280. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (log_operation) notice: Operation 'reboot' [23749] (call 2 from pacemaker-controld.12246) for host 'el8-a01n02' with device 'virsh_node2' returned: 0 (OK)
  281. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemakerd [12240] (pcmk_cpg_membership) info: Group pacemakerd event 2: el8-a01n02 (node 2 pid 1691) left via cluster exit
  282. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemakerd [12240] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline
  283. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemakerd [12240] (pcmk_cpg_membership) info: Group pacemakerd event 2: el8-a01n01 (node 1 pid 12240) is member
  284. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemakerd [12240] (pcmk_quorum_notification) info: Quorum retained | membership=117 members=1
  285. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemakerd [12240] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
  286. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (pcmk_cpg_membership) info: Group stonith-ng event 2: el8-a01n02 (node 2 pid 1693) left via cluster exit
  287. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline
  288. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
  289. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache
  290. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache
  291. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (pcmk_cpg_membership) info: Group stonith-ng event 2: el8-a01n01 (node 1 pid 12242) is member
  292. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-fenced [12242] (remote_op_done) notice: Operation 'reboot' targeting el8-a01n02 on el8-a01n01 for pacemaker-controld.12246@el8-a01n01.7224bb7a: OK
  293. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (pcmk_cpg_membership) info: Group cib event 2: el8-a01n02 (node 2 pid 1692) left via cluster exit
  294. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline
  295. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
  296. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache
  297. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache
  298. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (pcmk_cpg_membership) info: Group cib event 2: el8-a01n01 (node 1 pid 12241) is member
  299. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.61 2
  300. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.62 (null)
  301. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=62
  302. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=31, @rc-code=0, @op-status=0, @exec-time=2195
  303. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (pcmk_quorum_notification) info: Quorum retained | membership=117 members=1
  304. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
  305. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (peer_update_callback) info: Cluster node el8-a01n02 is now lost (was member)
  306. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/62, version=0.16.62)
  307. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n02 (node 2 pid 1695) left via cluster exit
  308. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline
  309. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_remove_voter) notice: Lost attribute writer el8-a01n02
  310. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_start_election_if_needed) info: Starting an election to determine the writer
  311. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
  312. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_remove) notice: Removing all el8-a01n02 attributes for peer loss
  313. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache
  314. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache
  315. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n01 (node 1 pid 12244) is member
  316. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/63)
  317. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (election_check) info: election-attrd won by local node
  318. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset)
  319. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/66)
  320. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) info: Transition 3 action 8 (hypervisor_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=31
  321. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a
  322. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/67)
  323. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-attrd [12244] (write_attribute) info: Sent CIB request 4 with 1 change for shutdown (id n/a, set n/a)
  324. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n02 (node 2 pid 1697) left via cluster exit
  325. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline
  326. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (peer_update_callback) info: Node el8-a01n02 is no longer a peer | DC=true old=0x4000000 new=0x0000000
  327. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (controld_delete_node_state) info: Deleting transient attributes for node el8-a01n02 (via CIB call 68) | xpath=//node_state[@uname='el8-a01n02']/transient_attributes
  328. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n01 (node 1 pid 12246) is member
  329. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (tengine_stonith_callback) notice: Stonith operation 2/5:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629: OK (0)
  330. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (tengine_stonith_callback) info: Stonith operation 2 for el8-a01n02 passed
  331. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (controld_delete_node_state) info: Deleting all state for node el8-a01n02 (via CIB call 71) | xpath=//node_state[@uname='el8-a01n02']/*
  332. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (tengine_stonith_notify) notice: Peer el8-a01n02 was terminated (reboot) by el8-a01n01 on behalf of pacemaker-controld.12246: OK | initiator=el8-a01n01 ref=7224bb7a-7dc4-44c1-a9ce-0e5491b6e52a
  333. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (controld_delete_node_state) info: Deleting all state for node el8-a01n02 (via CIB call 73) | xpath=//node_state[@uname='el8-a01n02']/*
  334. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.62 2
  335. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.63 (null)
  336. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=63
  337. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n01 | action 13
  338. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=peer_update_callback
  339. Jul 14 19:39:21 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/63, version=0.16.63)
  340. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/66, version=0.16.63)
  341. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.63 2
  342. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.64 (null)
  343. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=64
  344. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update, @expected=down
  345. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @in_ccm=false, @crm-debug-origin=post_cache_update, @expected=down
  346. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/67, version=0.16.64)
  347. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/4)
  348. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-execd [12243] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000
  349. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_lrm_rsc_op) info: Performing key=13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=drbd_stop_0
  350. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes to all (origin=local/crmd/68)
  351. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-execd [12243] (log_execute) info: executing - rsc:drbd action:stop call_id:33
  352. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/69)
  353. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/70)
  354. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/* to all (origin=local/crmd/71)
  355. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/72)
  356. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n01: Cancelled | call=25 key=drbd_monitor_60000 confirmed=true
  357. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/4, version=0.16.64)
  358. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: CIB update 4 result for shutdown: OK | rc=0
  359. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594755459
  360. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.64 2
  361. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.65 (null)
  362. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']
  363. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=65
  364. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n01/crmd/68, version=0.16.65)
  365. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.65 2
  366. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.66 (null)
  367. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=66
  368. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crmd=offline, @crm-debug-origin=peer_update_callback
  369. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/69, version=0.16.66)
  370. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.66 2
  371. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.67 (null)
  372. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=67
  373. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=send_stonith_update, @join=down
  374. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/70, version=0.16.67)
  375. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.67 2
  376. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.68 (null)
  377. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']
  378. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=68
  379. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/*: OK (rc=0, origin=el8-a01n01/crmd/71, version=0.16.68)
  380. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/72, version=0.16.68)
  381. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/* to all (origin=local/crmd/73)
  382. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/74)
  383. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-controld [12246] (cib_fencing_updated) info: Fencing update 70 for el8-a01n02: complete
  384. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-controld [12246] (cib_fencing_updated) info: Fencing update 72 for el8-a01n02: complete
  385. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/*: OK (rc=0, origin=el8-a01n01/crmd/73, version=0.16.68)
  386. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.68 2
  387. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.69 (null)
  388. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=69
  389. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
  390. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755562, @last-run=1594755562, @exec-time=0
  391. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/74, version=0.16.69)
  392. Jul 14 19:39:22 el8-a01n01.digimer.ca pacemaker-execd [12243] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1169
  393. Jul 14 19:39:27 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_ping) info: Reporting our current digest to el8-a01n01: c9b3b48b09509cbe32e9ec7efaa8bed9 for 0.16.69 (0x55f7b2a43cd0 0)
  394. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-execd [12243] (action_complete) notice: Giving up on drbd stop (rc=0): timeout (elapsed=99989ms, remaining=11ms)
  395. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_lrm_event) error: Result of stop operation for drbd on el8-a01n01: Timed Out | call=33 key=drbd_stop_0 timeout=100000ms
  396. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/75)
  397. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.69 2
  398. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.70 (null)
  399. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=70
  400. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=2:198;13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=33, @rc-code=198, @op-status=2, @exec-time=99989
  401. Jul 14 19:41:01 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_last_failure_0" operation_key="drbd_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629" transition-magic="2:198;13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629" exit-reason="" on_node="el8-a01n01" call-id="33" rc-code="
  402. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/75, version=0.16.70)
  403. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) notice: Transition 3 aborted by operation drbd_stop_0 'modify' on el8-a01n01: Event failed | magic=2:198;13:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629 cib=0.16.70 source=process_graph_event:453 complete=false
  404. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (update_failcount) info: Updating failcount for drbd on el8-a01n01 after failed stop: rc=198 (update=INFINITY, time=1594755662)
  405. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (process_graph_event) notice: Transition 3 action 13 (drbd_stop_0 on el8-a01n01): expected 'ok' but got 'OCF_TIMEOUT' | target-rc=0 rc=198 call-id=33
  406. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (run_graph) notice: Transition 3 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-warn-8.bz2): Complete
  407. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  408. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting fail-count-drbd#stop_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n01
  409. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (write_attribute) info: Sent CIB request 5 with 1 change for fail-count-drbd#stop_0 (id n/a, set n/a)
  410. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_peer_update) notice: Setting last-failure-drbd#stop_0[el8-a01n01]: (unset) -> 1594755662 | from el8-a01n01
  411. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/5)
  412. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (write_attribute) info: Sent CIB request 6 with 1 change for last-failure-drbd#stop_0 (id n/a, set n/a)
  413. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.70 2
  414. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.71 (null)
  415. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=71
  416. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-fail-count-drbd.stop_0" name="fail-count-drbd#stop_0" value="INFINITY"/>
  417. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/5, version=0.16.71)
  418. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/6)
  419. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: CIB update 5 result for fail-count-drbd#stop_0: OK | rc=0
  420. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: * fail-count-drbd#stop_0[el8-a01n01]=INFINITY
  421. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 3 aborted by status-1-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change | cib=0.16.71 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  422. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: --- 0.16.71 2
  423. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: Diff: +++ 0.16.72 (null)
  424. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: + /cib: @num_updates=72
  425. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-drbd.stop_0" name="last-failure-drbd#stop_0" value="1594755662"/>
  426. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/6, version=0.16.72)
  427. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (abort_transition_graph) info: Transition 3 aborted by status-1-last-failure-drbd.stop_0 doing create last-failure-drbd#stop_0=1594755662: Transient attribute change | cib=0.16.72 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  428. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is shutting down
  429. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_failure_0
  430. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_0
  431. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_fence_node) warning: Cluster node el8-a01n01 will be fenced: drbd:0 failed there
  432. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  433. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  434. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  435. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  436. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  437. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  438. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  439. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  440. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  441. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n01
  442. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n02 ]
  443. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node1 cannot run anywhere
  444. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node2 cannot run anywhere
  445. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:0 cannot run anywhere
  446. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  447. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  448. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:0 cannot run anywhere
  449. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (stage6) warning: Scheduling Node el8-a01n01 for STONITH
  450. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_stop_constraints) notice: Stop of failed resource drbd:0 is implicit after el8-a01n01 is fenced
  451. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Fence (reboot) el8-a01n01 'drbd:0 failed there'
  452. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node1 (Stopped)
  453. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node2 (Stopped)
  454. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:0 (Stopped)
  455. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:1 (Stopped)
  456. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave drbd:1 (Stopped)
  457. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  458. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) warning: Calculated transition 4 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-9.bz2
  459. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: CIB update 6 result for last-failure-drbd#stop_0: OK | rc=0
  460. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-attrd [12244] (attrd_cib_callback) info: * last-failure-drbd#stop_0[el8-a01n01]=1594755662
  461. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (handle_response) info: pe_calc calculation pe_calc-dc-1594755662-44 is obsolete
  462. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is shutting down
  463. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_failure_0
  464. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_0
  465. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_fence_node) warning: Cluster node el8-a01n01 will be fenced: drbd:0 failed there
  466. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  467. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  468. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  469. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  470. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  471. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  472. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  473. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  474. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  475. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n01
  476. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n02 ]
  477. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n01
  478. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  479. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n01
  480. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  481. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node1 cannot run anywhere
  482. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node2 cannot run anywhere
  483. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:0 cannot run anywhere
  484. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  485. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  486. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:0 cannot run anywhere
  487. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (stage6) warning: Scheduling Node el8-a01n01 for STONITH
  488. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_stop_constraints) notice: Stop of failed resource drbd:0 is implicit after el8-a01n01 is fenced
  489. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Fence (reboot) el8-a01n01 'drbd:0 failed there'
  490. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node1 (Stopped)
  491. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node2 (Stopped)
  492. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:0 (Stopped)
  493. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:1 (Stopped)
  494. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave drbd:1 (Stopped)
  495. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  496. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) warning: Calculated transition 5 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-10.bz2
  497. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (handle_response) info: pe_calc calculation pe_calc-dc-1594755662-45 is obsolete
  498. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (process_pe_message) info: Input has not changed since last time, not saving to disk
  499. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (determine_online_status) info: Node el8-a01n01 is shutting down
  500. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_failure_0
  501. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n01 at Jul 14 19:39:22 2020 | rc=198 id=drbd_last_0
  502. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_fence_node) warning: Cluster node el8-a01n01 will be fenced: drbd:0 failed there
  503. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  504. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  505. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 1 is already processed
  506. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (unpack_node_loop) info: Node 2 is already processed
  507. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  508. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  509. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  510. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  511. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (clone_print) info: Clone Set: drbd-clone [drbd]
  512. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n01
  513. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (short_print) info: Stopped: [ el8-a01n02 ]
  514. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n01
  515. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  516. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n01
  517. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  518. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node1 cannot run anywhere
  519. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource virsh_node2 cannot run anywhere
  520. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:0 cannot run anywhere
  521. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource hypervisor:1 cannot run anywhere
  522. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:1 cannot run anywhere
  523. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_color) info: Resource drbd:0 cannot run anywhere
  524. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (stage6) warning: Scheduling Node el8-a01n01 for STONITH
  525. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (native_stop_constraints) notice: Stop of failed resource drbd:0 is implicit after el8-a01n01 is fenced
  526. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogNodeActions) notice: * Fence (reboot) el8-a01n01 'drbd:0 failed there'
  527. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node1 (Stopped)
  528. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave virsh_node2 (Stopped)
  529. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:0 (Stopped)
  530. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave hypervisor:1 (Stopped)
  531. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogActions) info: Leave drbd:1 (Stopped)
  532. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  533. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-schedulerd[12245] (pcmk__log_transition_summary) warning: Calculated transition 6 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-10.bz2
  534. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  535. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (do_te_invoke) info: Processing graph 6 (ref=pe_calc-dc-1594755662-46) derived from /var/lib/pacemaker/pengine/pe-warn-10.bz2
  536. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-controld [12246] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n01 | action=2 timeout=60000
  537. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (handle_request) notice: Client pacemaker-controld.12246.215b449b wants to fence (reboot) 'el8-a01n01' with device '(any)'
  538. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n01 | id=13f82669-677a-40c7-b0e8-a5ebdd80ec6e state=0
  539. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node1 is eligible to fence (reboot) el8-a01n01: static-list
  540. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node2 is not eligible to fence (reboot) el8-a01n01: static-list
  541. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (process_remote_stonith_query) info: Query result 1 of 1 from el8-a01n01 for el8-a01n01/reboot (1 devices) 13f82669-677a-40c7-b0e8-a5ebdd80ec6e
  542. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (process_remote_stonith_query) info: All query replies have arrived, continuing (1 expected/1 received)
  543. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n01 for pacemaker-controld.12246|id=13f82669-677a-40c7-b0e8-a5ebdd80ec6e
  544. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n01 | for client pacemaker-controld.12246 (72s, 0s)
  545. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node1 is eligible to fence (reboot) el8-a01n01: static-list
  546. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (can_fence_host_with_device) notice: virsh_node2 is not eligible to fence (reboot) el8-a01n01: static-list
  547. Jul 14 19:41:02 el8-a01n01.digimer.ca pacemaker-fenced [12242] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n01'
  548. Jul 14 19:41:07 el8-a01n01.digimer.ca pacemaker-based [12241] (cib_process_ping) info: Reporting our current digest to el8-a01n01: fca1228c3b1aba61e67354286769aba4 for 0.16.72 (0x55f7b2a43cd0 0)
  549.  
  550. ==== Node 2
  551.  
  552. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemakerd [1691] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  553. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemakerd [1691] (pcmk_shutdown_worker) notice: Shutting down Pacemaker
  554. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemakerd [1691] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 1697
  555. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  556. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms
  557. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n01)
  558. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was (null))
  559. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594755459 | from el8-a01n01
  560. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (write_attribute) info: Sent CIB request 5 with 1 change for shutdown (id n/a, set n/a)
  561. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/5)
  562. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.44 2
  563. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.45 (null)
  564. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=45
  565. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/>
  566. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ <instance_attributes id="status-2">
  567. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ <nvpair id="status-2-shutdown" name="shutdown" value="1594755459"/>
  568. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ </instance_attributes>
  569. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ </transient_attributes>
  570. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.16.45)
  571. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: CIB update 5 result for shutdown: OK | rc=0
  572. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: * shutdown[el8-a01n02]=1594755459
  573. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (do_lrm_rsc_op) info: Performing key=9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=virsh_node2_stop_0
  574. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:27
  575. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) info: Result of monitor operation for virsh_node2 on el8-a01n02: Cancelled | call=22 key=virsh_node2_monitor_60000 confirmed=true
  576. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:27 exit-code:0 exec-time:1ms queue-time:0ms
  577. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/29)
  578. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000
  579. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (do_lrm_rsc_op) info: Performing key=14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=hypervisor_stop_0
  580. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (log_execute) info: executing - rsc:hypervisor action:stop call_id:29
  581. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n02: 0 (ok) | call=27 key=virsh_node2_stop_0 confirmed=true cib-update=31
  582. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n02: Cancelled | call=24 key=hypervisor_monitor_60000 confirmed=true
  583. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000
  584. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.45 2
  585. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.46 (null)
  586. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=46
  587. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0, @q
  588. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.16.46)
  589. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/30)
  590. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31)
  591. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.46 2
  592. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.47 (null)
  593. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=47
  594. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0
  595. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (do_lrm_rsc_op) info: Performing key=21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629 op=drbd_stop_0
  596. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (log_execute) info: executing - rsc:drbd action:stop call_id:31
  597. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n02: Cancelled | call=25 key=drbd_monitor_60000 confirmed=true
  598. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.16.47)
  599. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.47 2
  600. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.48 (null)
  601. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=48
  602. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;9:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=27, @rc-code=0, @op-status=0, @exec-time=1
  603. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.16.48)
  604. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/32)
  605. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.48 2
  606. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.49 (null)
  607. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=49
  608. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459, @exec-time=0
  609. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.16.49)
  610. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.49 2
  611. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.50 (null)
  612. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=50
  613. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755459, @last-run=1594755459
  614. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/52, version=0.16.50)
  615. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-controld [1697] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member)
  616. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594755459 | from el8-a01n01
  617. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (write_attribute) info: Sent CIB request 6 with 2 changes for shutdown (id n/a, set n/a)
  618. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/6)
  619. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.50 2
  620. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.51 (null)
  621. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=51
  622. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
  623. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ <instance_attributes id="status-1">
  624. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ <nvpair id="status-1-shutdown" name="shutdown" value="1594755459"/>
  625. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ </instance_attributes>
  626. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ </transient_attributes>
  627. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.16.51)
  628. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: CIB update 6 result for shutdown: OK | rc=0
  629. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594755459
  630. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: * shutdown[el8-a01n02]=1594755459
  631. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1022
  632. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-execd [1694] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1023
  633. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.51 2
  634. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.52 (null)
  635. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=52
  636. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;10:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=26, @rc-code=0, @op-status=0, @exec-time=704
  637. Jul 14 19:37:39 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/53, version=0.16.52)
  638. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n02: 0 (ok) | call=29 key=hypervisor_stop_0 confirmed=true cib-update=33
  639. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/33)
  640. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.52 2
  641. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.53 (null)
  642. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=53
  643. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;14:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2253
  644. Jul 14 19:37:41 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.16.53)
  645. Jul 14 19:37:46 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 12bcdc96a01db61039f2acac78431a32 for 0.16.53 (0x56106dec6080 0)
  646.  
  647. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-execd [1694] (action_complete) notice: Giving up on drbd stop (rc=0): timeout (elapsed=99989ms, remaining=11ms)
  648. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-controld [1697] (process_lrm_event) error: Result of stop operation for drbd on el8-a01n02: Timed Out | call=31 key=drbd_stop_0 timeout=100000ms
  649. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/34)
  650. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.53 2
  651. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.54 (null)
  652. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=54
  653. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=2:198;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=31, @rc-code=198, @op-status=2, @exec-time=99989
  654. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_last_failure_0" operation_key="drbd_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629" transition-magic="2:198;21:1:0:25b0a419-93c5-45d0-9b5d-f240441d9629" exit-reason="" on_node="el8-a01n02" call-id="31" rc-code="1
  655. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.16.54)
  656. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_peer_update) notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
  657. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (write_attribute) info: Sent CIB request 7 with 1 change for fail-count-drbd#stop_0 (id n/a, set n/a)
  658. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/7)
  659. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_peer_update) notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594755559 | from el8-a01n01
  660. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (write_attribute) info: Sent CIB request 8 with 1 change for last-failure-drbd#stop_0 (id n/a, set n/a)
  661. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/8)
  662. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.54 2
  663. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.55 (null)
  664. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=55
  665. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-fail-count-drbd.stop_0" name="fail-count-drbd#stop_0" value="INFINITY"/>
  666. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.16.55)
  667. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: CIB update 7 result for fail-count-drbd#stop_0: OK | rc=0
  668. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: * fail-count-drbd#stop_0[el8-a01n02]=INFINITY
  669. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.55 2
  670. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.56 (null)
  671. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=56
  672. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-drbd.stop_0" name="last-failure-drbd#stop_0" value="1594755559"/>
  673. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/8, version=0.16.56)
  674. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: CIB update 8 result for last-failure-drbd#stop_0: OK | rc=0
  675. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-attrd [1695] (attrd_cib_callback) info: * last-failure-drbd#stop_0[el8-a01n02]=1594755559
  676. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.56 2
  677. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.57 (null)
  678. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=57
  679. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  680. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.16.57)
  681. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.57 2
  682. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.58 (null)
  683. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=58
  684. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  685. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.16.58)
  686. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.58 2
  687. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.59 (null)
  688. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=59
  689. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;6:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=28, @rc-code=0, @op-status=0
  690. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/59, version=0.16.59)
  691. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.59 2
  692. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.60 (null)
  693. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=60
  694. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;7:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=29, @rc-code=0, @op-status=0, @exec-time=1
  695. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/60, version=0.16.60)
  696. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: --- 0.16.60 2
  697. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: Diff: +++ 0.16.61 (null)
  698. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib: @num_updates=61
  699. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @transition-magic=-1:193;8:3:0:25b0a419-93c5-45d0-9b5d-f240441d9629, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594755559, @last-run=1594755559, @exec-time=0
  700. Jul 14 19:39:19 el8-a01n02.digimer.ca pacemaker-based [1692] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.16.61)
  701. client_loop: send disconnect: Broken pipe
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!