Jun 26 13:37:51 an-c03n01 corosync[964]:  [TOTEM ] A processor failed, forming new configuration.
Jun 26 13:37:53 an-c03n01 corosync[964]:  [QUORUM] This node is within the non-primary component and will NOT provide any services.
Jun 26 13:37:53 an-c03n01 corosync[964]:  [QUORUM] Members[1]: 1
Jun 26 13:37:53 an-c03n01 corosync[964]:  [QUORUM] Members[1]: 1
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: pcmk_quorum_notification: Membership 28: quorum lost (1)
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node an-c03n02.alteeve.ca[2] - state is now lost (was member)
Jun 26 13:37:53 an-c03n01 crmd[987]:  warning: match_down_event: No match for shutdown action on 2
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: peer_update_callback: Stonith/shutdown of an-c03n02.alteeve.ca not matched
Jun 26 13:37:53 an-c03n01 corosync[964]:  [TOTEM ] A processor joined or left the membership and a new membership (10.20.30.1:28) was formed.
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 26 13:37:53 an-c03n01 crmd[987]:  warning: match_down_event: No match for shutdown action on 2
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: peer_update_callback: Stonith/shutdown of an-c03n02.alteeve.ca not matched
Jun 26 13:37:53 an-c03n01 corosync[964]:  [MAIN  ] Completed service synchronization, ready to provide service.
Jun 26 13:37:53 an-c03n01 attrd[985]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Jun 26 13:37:53 an-c03n01 attrd[985]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jun 26 13:37:53 an-c03n01 pengine[986]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: pe_fence_node: Node an-c03n02.alteeve.ca will be fenced because the node is no longer part of the cluster
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: determine_online_status: Node an-c03n02.alteeve.ca is unclean
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_ipmi_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_ipmi_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_psu1_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_psu1_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_psu2_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: custom_action: Action fence_n01_psu2_stop_0 on an-c03n02.alteeve.ca is unrunnable (offline)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: stage6: Scheduling Node an-c03n02.alteeve.ca for STONITH
Jun 26 13:37:53 an-c03n01 pengine[986]:   notice: LogActions: Stop    fence_n01_ipmi#011(an-c03n02.alteeve.ca)
Jun 26 13:37:53 an-c03n01 pengine[986]:   notice: LogActions: Stop    fence_n01_psu1#011(an-c03n02.alteeve.ca)
Jun 26 13:37:53 an-c03n01 pengine[986]:   notice: LogActions: Stop    fence_n01_psu2#011(an-c03n02.alteeve.ca)
Jun 26 13:37:53 an-c03n01 pengine[986]:  warning: process_pe_message: Calculated Transition 15: /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jun 26 13:37:53 an-c03n01 crmd[987]:   notice: te_fence_node: Executing reboot fencing operation (14) on an-c03n02.alteeve.ca (timeout=60000)
Jun 26 13:37:53 an-c03n01 stonith-ng[983]:   notice: handle_request: Client crmd.987.0357e58d wants to fence (reboot) 'an-c03n02.alteeve.ca' with device '(any)'
Jun 26 13:37:53 an-c03n01 stonith-ng[983]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for an-c03n02.alteeve.ca: 277f7416-b2e8-48f7-8095-52e4dff67cd8 (0)
Jun 26 13:38:32 an-c03n01 stonith-ng[983]:    error: log_operation: Operation 'reboot' [8482] (call 3 from crmd.987) for host 'an-c03n02.alteeve.ca' with device 'fence_n02_ipmi' returned: -201 (Generic Pacemaker error)
Jun 26 13:38:32 an-c03n01 stonith-ng[983]:  warning: log_operation: fence_n02_ipmi:8482 [ Rebooting machine @ IPMI:an-c03n02.ipmi...Failed ]
Jun 26 13:38:32 an-c03n01 stonith-ng[983]:   notice: process_remote_stonith_exec: Call to fence_n02_ipmi for an-c03n02.alteeve.ca on behalf of crmd.987@an-c03n01.alteeve.ca: Generic Pacemaker error (-201)
Jun 26 13:38:32 an-c03n01 fence_apc_snmp: Parse error: Ignoring unknown option 'nodename=an-c03n02.alteeve.ca
Jun 26 13:38:33 an-c03n01 stonith-ng[983]:   notice: log_operation: Operation 'reboot' [8718] (call 3 from crmd.987) for host 'an-c03n02.alteeve.ca' with device 'fence_n02_psu1' returned: 0 (OK)
Jun 26 13:38:33 an-c03n01 stonith-ng[983]:   notice: process_remote_stonith_exec: Call to fence_n02_psu1 for an-c03n02.alteeve.ca on behalf of crmd.987@an-c03n01.alteeve.ca: OK (0)
Jun 26 13:38:33 an-c03n01 fence_apc_snmp: Parse error: Ignoring unknown option 'nodename=an-c03n02.alteeve.ca
Jun 26 13:38:34 an-c03n01 stonith-ng[983]:   notice: log_operation: Operation 'reboot' [8755] (call 3 from crmd.987) for host 'an-c03n02.alteeve.ca' with device 'fence_n02_psu2' returned: 0 (OK)
Jun 26 13:38:34 an-c03n01 stonith-ng[983]:   notice: process_remote_stonith_exec: Call to fence_n02_psu2 for an-c03n02.alteeve.ca on behalf of crmd.987@an-c03n01.alteeve.ca: OK (0)
Jun 26 13:38:34 an-c03n01 stonith-ng[983]:   notice: remote_op_done: Operation reboot of an-c03n02.alteeve.ca by an-c03n01.alteeve.ca for crmd.987@an-c03n01.alteeve.ca.277f7416: OK
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: tengine_stonith_callback: Stonith operation 3/14:15:0:ec97930a-6ce6-443c-9eb0-dd953b626ebc: OK (0)
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: tengine_stonith_notify: Peer an-c03n02.alteeve.ca was terminated (reboot) by an-c03n01.alteeve.ca for an-c03n01.alteeve.ca: OK (ref=277f7416-b2e8-48f7-8095-52e4dff67cd8) by client crmd.987
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: run_graph: Transition 15 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 26 13:38:34 an-c03n01 pengine[986]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Jun 26 13:38:34 an-c03n01 pengine[986]:   notice: process_pe_message: Calculated Transition 16: /var/lib/pacemaker/pengine/pe-input-13.bz2
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: run_graph: Transition 16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-13.bz2): Complete
Jun 26 13:38:34 an-c03n01 crmd[987]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]