[root@el8-a01n01 ~]# cat /var/log/pacemaker/pacemaker.log
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_log_init) 	info: Cannot change active directory to /var/lib/pacemaker/cores: No such file or directory (2)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (get_cluster_type) 	info: Detected an active 'corosync' cluster
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (mcp_read_config) 	info: Reading configure for stack: corosync
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (main) 	notice: Starting Pacemaker 2.0.3-5.el8_2.1 | build=4b1f869f0f features: generated-manpages agent-manpages ncurses libqb-logging libqb-ipc systemd nagios  corosync-native atomic-attrd acls
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (main) 	info: Maximum core file size is: 18446744073709551615
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (qb_ipcs_us_publish) 	info: server name: pacemakerd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to lrmd IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to cib_ro IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to crmd IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to attrd IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to pengine IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk__ipc_is_authentic_process_active) 	info: Could not connect to stonith-ng IPC: Connection refused
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Created entry 724e6ee6-fc51-496f-b6d2-093b6b017a7b/0x55dbca2ebf90 for node el8-a01n01/1 (1 total)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Node 1 is now known as el8-a01n01
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Node 1 has uuid 1
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_update_peer_proc) 	info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (cluster_connect_quorum) 	warning: Quorum lost
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Created entry c7daba3d-a73b-4929-9d26-7ec6ef2a18a7/0x55dbca4ed8b0 for node el8-a01n02/2 (2 total)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Node 2 is now known as el8-a01n02
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_get_peer) 	info: Node 2 has uuid 2
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Using uid=189 and group=189 for process pacemaker-based
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375409 for process pacemaker-based
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375410 for process pacemaker-fenced
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375411 for process pacemaker-execd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Using uid=189 and group=189 for process pacemaker-attrd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375412 for process pacemaker-attrd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Using uid=189 and group=189 for process pacemaker-schedulerd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375413 for process pacemaker-schedulerd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Using uid=189 and group=189 for process pacemaker-controld
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (start_child) 	info: Forked child 375414 for process pacemaker-controld
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (main) 	notice: Pacemaker daemon successfully started and accepting connections
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_quorum_notification) 	info: Quorum still lost | membership=140 members=1
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_update_peer_state_iter) 	notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_reap_unseen_nodes) 	info: State of node el8-a01n02[2] is still unknown
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_cpg_membership) 	info: Group pacemakerd event 0: node 1 pid 375401 joined via cpg_join
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_cpg_membership) 	info: Group pacemakerd event 0: el8-a01n01 (node 1 pid 375401) is member
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (main) 	notice: Starting Pacemaker node attribute manager
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (main) 	notice: Starting Pacemaker CIB manager
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (get_cluster_type) 	info: Verifying cluster type: 'corosync'
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (get_cluster_type) 	info: Assuming an active 'corosync' cluster
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk__daemon_can_write) 	notice: /var/lib/pacemaker/cib/cib.xml not found: No such file or directory
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk__daemon_can_write) 	notice: /var/lib/pacemaker/cib/cib.xml.sig not found: No such file or directory
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (retrieveCib) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_read_and_verify) 	warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (readCibXmlFile) 	warning: Primary configuration corrupt or unusable, trying backups in /var/lib/pacemaker/cib
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (readCibXmlFile) 	warning: Continuing with an empty configuration.
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd     [375411] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd     [375411] (main) 	notice: Starting Pacemaker local executor
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_ipcs_us_publish) 	info: server name: lrmd
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd     [375411] (main) 	notice: Pacemaker local executor successfully started and accepting connections
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_cluster_connect) 	notice: Connecting to cluster infrastructure: corosync
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld  [375414] (main) 	notice: Starting Pacemaker controller
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_log) 	info: Input I_STARTUP received in state S_STARTING from crmd_init
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (main) 	notice: Starting Pacemaker fencer
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (get_cluster_type) 	info: Verifying cluster type: 'corosync'
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (get_cluster_type) 	info: Assuming an active 'corosync' cluster
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_cluster_connect) 	notice: Connecting to cluster infrastructure: corosync
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Created entry 64b0555c-0df6-469b-b304-b2214c0c1b76/0x555fc5d7ed60 for node el8-a01n01/1 (1 total)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Node 1 is now known as el8-a01n01
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (crm_log_init) 	info: Changed active directory to /var/lib/pacemaker/cores
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (main) 	notice: Starting Pacemaker scheduler
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (qb_ipcs_us_publish) 	info: server name: pengine
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (main) 	notice: Pacemaker scheduler successfully started and accepting connections
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Created entry 1b70802f-7e5a-434f-8a6a-5c3b48769ac1/0x555a63b7b7a0 for node el8-a01n01/1 (1 total)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Node 1 is now known as el8-a01n01
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Node 1 has uuid 1
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_update_peer_proc) 	info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_update_peer_state_iter) 	notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (init_cs_connection_once) 	info: Connection to 'corosync': established
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (qb_ipcs_us_publish) 	info: server name: cib_ro
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (qb_ipcs_us_publish) 	info: server name: cib_rw
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (qb_ipcs_us_publish) 	info: server name: cib_shm
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (main) 	notice: Pacemaker CIB manager successfully started and accepting connections
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk_cpg_membership) 	info: Group cib event 0: node 1 pid 375409 joined via cpg_join
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk_cpg_membership) 	info: Group cib event 0: el8-a01n01 (node 1 pid 375409) is member
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Node 1 has uuid 1
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_update_peer_proc) 	info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_update_peer_state_iter) 	notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (init_cs_connection_once) 	info: Connection to 'corosync': established
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_read_and_verify) 	warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (setup_cib) 	info: Watching for stonith topology changes
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (qb_ipcs_us_publish) 	info: server name: stonith-ng
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (main) 	notice: Pacemaker fencer successfully started and accepting connections
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (pcmk_cpg_membership) 	info: Group stonith-ng event 0: node 1 pid 375410 joined via cpg_join
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (pcmk_cpg_membership) 	info: Group stonith-ng event 0: el8-a01n01 (node 1 pid 375410) is member
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (init_cib_cache_cb) 	info: Updating device list from the cib: init
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.0.0
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (unpack_nodes) 	info: Creating a fake local node
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.0.0 of the CIB to disk (digest: 2469c254d4b2872e864ff6b9c730ac0c)
Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.azN5iO (digest: /var/lib/pacemaker/cib/cib.b8uEay)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk_cpg_membership) 	info: Group cib event 1: node 2 pid 4565 joined via cluster join
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (pcmk_cpg_membership) 	info: Group stonith-ng event 1: node 2 pid 4566 joined via cluster join
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (pcmk_cpg_membership) 	info: Group stonith-ng event 1: el8-a01n01 (node 1 pid 375410) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_cpg_membership) 	info: Group pacemakerd event 1: node 2 pid 4564 joined via cluster join
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_cpg_membership) 	info: Group pacemakerd event 1: el8-a01n01 (node 1 pid 375401) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_cpg_membership) 	info: Group pacemakerd event 1: el8-a01n02 (node 2 pid 4564) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_update_peer_proc) 	info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk_cpg_membership) 	info: Group cib event 1: el8-a01n01 (node 1 pid 375409) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_cs_flush) 	info: Sent 0 CPG messages  (1 remaining, last=7): Try again (6)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_quorum_notification) 	notice: Quorum acquired | membership=144 members=2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_update_peer_state_iter) 	notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Created entry 985cf59e-a073-4bb7-8a71-1afd8108238b/0x555a63b823f0 for node el8-a01n02/2 (2 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Node 2 is now known as el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_get_peer) 	info: Node 2 has uuid 2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (pcmk_cpg_membership) 	info: Group cib event 1: el8-a01n02 (node 2 pid 4565) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_update_peer_proc) 	info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (crm_update_peer_state_iter) 	notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Created entry a122cdea-22b5-4c4d-89a1-fc060ffad198/0x555fc5d7c270 for node el8-a01n02/2 (2 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Node 2 is now known as el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_get_peer) 	info: Node 2 has uuid 2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (pcmk_cpg_membership) 	info: Group stonith-ng event 1: el8-a01n02 (node 2 pid 4566) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_update_peer_proc) 	info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (crm_update_peer_state_iter) 	notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n02/attrd/2, version=0.0.0)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (main) 	info: CIB connection active
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (get_cluster_type) 	info: Verifying cluster type: 'corosync'
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (get_cluster_type) 	info: Assuming an active 'corosync' cluster
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_cluster_connect) 	notice: Connecting to cluster infrastructure: corosync
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Created entry 521b1756-729a-45ed-a585-3cbafa292175/0x55f3182f8a20 for node el8-a01n01/1 (1 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Node 1 is now known as el8-a01n01
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Node 1 has uuid 1
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_update_peer_proc) 	info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_update_peer_state_iter) 	notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (init_cs_connection_once) 	info: Connection to 'corosync': established
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (main) 	info: Cluster connection active
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_erase_attrs) 	info: Clearing transient attributes from CIB | xpath=//node_state[@uname='el8-a01n01']/transient_attributes
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (qb_ipcs_us_publish) 	info: server name: attrd
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (main) 	notice: Pacemaker node attribute manager successfully started and accepting connections
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (pcmk_cpg_membership) 	info: Group attrd event 0: node 1 pid 375412 joined via cpg_join
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (pcmk_cpg_membership) 	info: Group attrd event 0: el8-a01n01 (node 1 pid 375412) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.0.0 2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.1.0 (null)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=1
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/nodes:  <node id="1" uname="el8-a01n01"/>
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/nodes:  <node id="2" uname="el8-a01n02"/>
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/4, version=0.1.0)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes to all (origin=local/attrd/2)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Created entry 2162ef55-efca-47dc-95c3-b15f8ac5669b/0x55f3182fb6d0 for node el8-a01n02/2 (2 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Node 2 is now known as el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_get_peer) 	info: Node 2 has uuid 2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (pcmk_cpg_membership) 	info: Group attrd event 0: el8-a01n02 (node 2 pid 4568) is member
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_update_peer_proc) 	info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (crm_update_peer_state_iter) 	notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_check_for_new_writer) 	notice: Recorded new attribute writer: el8-a01n02 (was unset)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_message) 	info: Processing sync-response from el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2 | from el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2 | from el8-a01n01
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.1.0 of the CIB to disk (digest: 7aefab66c16403ca8289ab3d2d263d82)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nXt7xa (digest: /var/lib/pacemaker/cib/cib.M3TlhX)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes: OK (rc=0, origin=el8-a01n01/attrd/2, version=0.1.0)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (get_cluster_type) 	info: Verifying cluster type: 'corosync'
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (get_cluster_type) 	info: Assuming an active 'corosync' cluster
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_cluster_connect) 	notice: Connecting to cluster infrastructure: corosync
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Created entry c7a1c8a9-639f-4423-9092-8fc91d5806e6/0x558f70425eb0 for node el8-a01n01/1 (1 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Node 1 is now known as el8-a01n01
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Cluster node el8-a01n01 is now in unknown state
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Node 1 has uuid 1
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_proc) 	info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Node el8-a01n01 is now a peer | DC=<none> old=0x0000000 new=0x4000000
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_trigger_stonith_history_sync) 	info: Fence history will be synchronized cluster-wide within 5 seconds
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (init_cs_connection_once) 	info: Connection to 'corosync': established
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (cluster_connect_quorum) 	notice: Quorum acquired
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Created entry 6a74f85b-f24c-49ec-acac-ebc0c3e8ad7d/0x558f70428740 for node el8-a01n02/2 (2 total)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Node 2 is now known as el8-a01n02
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Cluster node el8-a01n02 is now in unknown state
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_get_peer) 	info: Node 2 has uuid 2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_ha_control) 	info: Connected to the cluster
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (lrmd_ipc_connect) 	info: Connecting to executor
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (crm_client_new) 	debug: Connecting 0x556a6a524fb0 for uid=189 gid=189 pid=375414 id=09acd970-a9ad-4616-9dfb-40686968c8db
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (handle_new_connection) 	debug: IPC credentials authenticated (/dev/shm/qb-375411-375414-9-4xxB6U/qb)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_ipcs_shm_connect) 	debug: connecting to client [375414]
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/4)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/4, version=0.1.0)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_control) 	info: Connection to the executor established
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_started) 	info: Delaying start, no membership data (0000000000100000)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_started) 	info: Delaying start, no membership data (0000000000100000)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (pcmk_quorum_notification) 	info: Quorum retained | membership=144 members=2
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_state_iter) 	notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Cluster node el8-a01n01 is now member (was in unknown state)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_state_iter) 	notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Cluster node el8-a01n02 is now member (was in unknown state)
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (qb_ipcs_us_publish) 	info: server name: crmd
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_started) 	notice: Pacemaker controller successfully started and accepting connections
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_log) 	info: Input I_PENDING received in state S_STARTING from do_started
Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (pcmk_cpg_membership) 	info: Group crmd event 0: node 1 pid 375414 joined via cpg_join
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (pcmk_cpg_membership) 	info: Group crmd event 0: el8-a01n01 (node 1 pid 375414) is member
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (pcmk_cpg_membership) 	info: Group crmd event 0: el8-a01n02 (node 2 pid 4570) is member
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_proc) 	info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (peer_update_callback) 	info: Node el8-a01n02 is now a peer | DC=<none> old=0x0000000 new=0x4000000
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_trigger_stonith_history_sync) 	info: Fence history will be synchronized cluster-wide within 5 seconds
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_trigger_stonith_history_sync) 	info: Fence history will be synchronized cluster-wide within 30 seconds
Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_connect_stonith) 	notice: Fencer successfully connected
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.1.0 2
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.2.0 (null)
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=2
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <primitive class="stonith" id="virsh_node1" type="fence_virsh"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <instance_attributes id="virsh_node1-instance_attributes">
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-delay" name="delay" value="15"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-ipaddr" name="ipaddr" value="10.20.255.250"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-login" name="login" value="root"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-password_script" name="password_script" value="/root/stonith.pw"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n01"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node1-instance_attributes-port" name="port" value="el8-a01n01"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </instance_attributes>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <operations>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="virsh_node1-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </operations>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </primitive>
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.2.0)
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.2.0
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.2.0 of the CIB to disk (digest: 0d075a040dee76aba3d9bb362ddfe793)
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OutjvV (digest: /var/lib/pacemaker/cib/cib.cbWZZA)
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (action_synced_wait) 	info: Managed fence_virsh_metadata_1 process 376630 exited with rc=0
Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_device_register) 	notice: Added 'virsh_node1' to the device list (1 active devices)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (election_count_vote) 	info: election-DC round 1 (owner node ID 2) pass: vote from el8-a01n02 (Uptime)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_PENDING -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (election_check) 	info: election-DC won by local node
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_log) 	info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_control) 	info: Registering TE UUID: 0df6240b-f9a0-4c7b-bf60-fd2dab145e11
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (set_graph_functions) 	info: Setting custom graph functions
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_dc_takeover) 	info: Taking over DC status for this partition
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_readwrite) 	info: We are now in R/W mode
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.2.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/8)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/8, version=0.2.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/10)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.2.0 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.3.0 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=3
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/crm_config:  <cluster_property_set id="cib-bootstrap-options"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                   <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                 </cluster_property_set>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/10, version=0.3.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/12)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.3.0 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.4.0 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=4
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']:  <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.0.3-5.el8_2.1-4b1f869f0f"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/12, version=0.4.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/14)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.4.0 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.5.0 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=5
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']:  <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/14, version=0.5.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (join_make_offer) 	info: Making join-1 offers based on membership event 144
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (join_make_offer) 	info: Sending join-1 offer to el8-a01n01
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (join_make_offer) 	info: Sending join-1 offer to el8-a01n02
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_dc_join_offer_all) 	info: Waiting on join-1 requests from 2 outstanding nodes
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/16)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.5.0 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.0 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=6
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']:  <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="el8-anvil-01"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_dc) 	info: Set DC to el8-a01n01 (3.3.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_expected) 	info: update_dc: Node el8-a01n01[1] - expected state is now member (was (null))
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_expected) 	info: do_dc_join_filter_offer: Node el8-a01n02[2] - expected state is now member (was (null))
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/16, version=0.6.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (controld_delete_node_state) 	info: Deleting resource history for node el8-a01n01 (via CIB call 24) | xpath=//node_state[@uname='el8-a01n01']/lrm
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_replace) 	info: Digest matched on replace from el8-a01n01: dc9656c0a868cfbd5503dbb3659ae0b0
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_replace) 	info: Replaced 0.6.0 with 0.6.0 from el8-a01n01
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (controld_delete_node_state) 	info: Deleting resource history for node el8-a01n02 (via CIB call 26) | xpath=//node_state[@uname='el8-a01n02']/lrm
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_replace operation for section 'all': OK (rc=0, origin=el8-a01n01/crmd/21, version=0.6.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/22)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/23)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm to all (origin=local/crmd/24)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/25)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm to all (origin=local/crmd/26)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/22, version=0.6.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/23, version=0.6.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm: OK (rc=0, origin=el8-a01n01/crmd/24, version=0.6.0)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.0 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.1 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=1
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status:  <node_state id="1" uname="el8-a01n01" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                 <lrm id="1">
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                   <lrm_resources/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                 </lrm>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++               </node_state>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/25, version=0.6.1)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm: OK (rc=0, origin=el8-a01n01/crmd/26, version=0.6.1)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/30)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.1 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.2 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status:  <node_state id="2" uname="el8-a01n02" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                 <lrm id="2">
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                   <lrm_resources/>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                 </lrm>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++               </node_state>
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/27, version=0.6.2)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/30, version=0.6.2)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.2 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.3 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=3
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_state_transition, @join=member, @expected=member
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_state_transition, @join=member, @expected=member
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/31, version=0.6.3)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/32)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.3 2
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.4 (null)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=4, @have-quorum=1, @dc-uuid=1
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/32, version=0.6.4)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.3.0 of the CIB to disk (digest: be8b332ecdd2a858160b9164e32749f2)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3vARTJ (digest: /var/lib/pacemaker/cib/cib.4ahR0w)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-4.raw
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.6.0 of the CIB to disk (digest: 90d9c1475d4bcd1028ad0c0f53e4d2ee)
Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HAXpWJ (digest: /var/lib/pacemaker/cib/cib.yoEE5w)
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node1 on el8-a01n01
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      virsh_node1     ( el8-a01n01 )  
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-0.bz2
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 0 (ref=pe_calc-dc-1594762149-9) derived from /var/lib/pacemaker/pengine/pe-input-0.bz2
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n02 | action 2
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n01 | action 1
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'virsh_node1' not in cache
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_rsc_register) 	info: Cached agent information for 'virsh_node1'
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_monitor_0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.4 2
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.5 (null)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=5
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_update_resource
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources:  <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-sta
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/12, version=0.6.5)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.5 2
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.6 (null)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=6
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:7;2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=5, @rc-code=7, @op-status=0, @last-rc-change=1594762150, @last-run=1594762150, @exec-time=12
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/13, version=0.6.6)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (action_synced_wait) 	info: Managed fence_virsh_metadata_1 process 376820 exited with rc=0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=5, reply=1, notify=0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:virsh_node1 action:monitor call_id:5
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (stonith_api_signon) 	debug: Attempting fencer connection by pacemaker-execd with mainloop
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 0 action 2 (virsh_node1_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=5
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/35)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.6 2
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.7 (null)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=7
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_update_resource
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-sta
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/35, version=0.6.7)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (qb_rb_open_2) 	debug: shm size:131085; real_size:135168; rb->word_size:33792
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (stonith_api_signon) 	debug: Connection to fencer by pacemaker-execd succeeded (registration token: d4ea23c3-5a74-4ad3-99e3-ae802a8f64a1)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:virsh_node1 action:monitor call_id:5  exit-code:7 exec-time:6ms queue-time:0ms
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of probe operation for virsh_node1 on el8-a01n01: 7 (not running) | call=5 key=virsh_node1_monitor_0 confirmed=true cib-update=36
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.7 2
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.8 (null)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=8
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:7;1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=5, @rc-code=7, @op-status=0, @last-rc-change=1594762150, @last-run=1594762150, @exec-time=6
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/36, version=0.6.8)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 0 action 1 (virsh_node1_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=5
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation virsh_node1_start_0 locally on el8-a01n01 | action 3
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_start_0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=6, reply=1, notify=0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:virsh_node1 action:start call_id:6
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/37)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.8 2
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.6.9 (null)
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=9
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @operation_key=virsh_node1_start_0, @operation=start, @transition-key=3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/37, version=0.6.9)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.6.9 2
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.0 (null)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=7, @num_updates=0
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <primitive class="stonith" id="virsh_node2" type="fence_virsh"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <instance_attributes id="virsh_node2-instance_attributes">
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node2-instance_attributes-ipaddr" name="ipaddr" value="10.20.255.250"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node2-instance_attributes-login" name="login" value="root"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node2-instance_attributes-password_script" name="password_script" value="/root/stonith.pw"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n02"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="virsh_node2-instance_attributes-port" name="port" value="el8-a01n02"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </instance_attributes>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <operations>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="virsh_node2-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </operations>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </primitive>
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.7.0
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 0 aborted by primitive.virsh_node2 'create': Configuration change | cib=0.7.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_device_register) 	notice: Added 'virsh_node2' to the device list (2 active devices)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.7.0)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-5.raw
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.7.0 of the CIB to disk (digest: ed4816be9cba1c0fb4b22dfa9b7cb0b2)
Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.kG2PCG (digest: /var/lib/pacemaker/cib/cib.lhehFG)
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:376837:stderr [ 2020-07-14 21:29:15,156 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:376837:stderr [  ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:376837:stderr [  ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[376837] stderr: [ 2020-07-14 21:29:15,156 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[376837] stderr: [  ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[376837] stderr: [  ]
Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (monitor). remaining timeout is 15
Jul 14 21:29:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_procfs_pid_of) 	info: Found pacemaker-based active as process 375409
Jul 14 21:29:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (throttle_send_command) 	info: New throttle mode: negligible load (was undetermined)
Jul 14 21:29:18 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: 38bdea493108431f663dd4ee0010b3ce for 0.7.0 (0x555a63c2a960 0)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:377243:stderr [ 2020-07-14 21:29:21,392 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:377243:stderr [  ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:377243:stderr [  ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377243] stderr: [ 2020-07-14 21:29:21,392 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377243] stderr: [  ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377243] stderr: [  ]
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (monitor) the maximum number of times (2) allowed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	notice: Operation 'monitor' [377243] for device 'virsh_node1' returned: -201 (Generic Pacemaker error)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	info: finished - rsc:virsh_node1 action:start call_id:6  exit-code:1 exec-time:11453ms queue-time:1ms
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of start operation for virsh_node1 on el8-a01n01: 1 (error) | call=6 key=virsh_node1_start_0 confirmed=true cib-update=38
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/38)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.0 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.1 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=6, @rc-code=1, @op-status=0, @exec-time=11453, @queue-time=1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']:  <lrm_rsc_op id="virsh_node1_last_failure_0" operation_key="virsh_node1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" ca
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 0 aborted by operation virsh_node1_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.7.1 source=process_graph_event:453 complete=false
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_failcount) 	info: Updating failcount for virsh_node1 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762161)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/38, version=0.7.1)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	notice: Transition 0 action 3 (virsh_node1_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=6
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 0 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-0.bz2): Complete
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting fail-count-virsh_node1#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting last-failure-virsh_node1#start_0[el8-a01n01]: (unset) -> 1594762161 | from el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.1 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.2 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']:  <transient_attributes id="1"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                     <instance_attributes id="status-1">
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                       <nvpair id="status-1-fail-count-virsh_node1.start_0" name="fail-count-virsh_node1#start_0" value="INFINITY"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                     </instance_attributes>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                   </transient_attributes>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/4, version=0.7.2)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.2 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.3 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=3
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-last-failure-virsh_node1.start_0" name="last-failure-virsh_node1#start_0" value="1594762161"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.7.3)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 0 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.7.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=true
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 0 aborted by status-1-last-failure-virsh_node1.start_0 doing create last-failure-virsh_node1#start_0=1594762161: Transient attribute change | cib=0.7.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	 FAILED el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node1 on el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node2 on el8-a01n02
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Recover    virsh_node1     ( el8-a01n01 )  
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      virsh_node2     ( el8-a01n02 )  
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-1.bz2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_response) 	info: pe_calc calculation pe_calc-dc-1594762161-14 is obsolete
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	 FAILED el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node1 on el8-a01n02
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node2 on el8-a01n01
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Recover    virsh_node1     ( el8-a01n01 -> el8-a01n02 )  
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      virsh_node2     (               el8-a01n01 )  
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 2 (ref=pe_calc-dc-1594762161-15) derived from /var/lib/pacemaker/pengine/pe-input-2.bz2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation virsh_node1_stop_0 locally on el8-a01n01 | action 1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_stop_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=7, reply=1, notify=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:virsh_node1 action:stop call_id:7
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	info: finished - rsc:virsh_node1 action:stop call_id:7  exit-code:0 exec-time:1ms queue-time:0ms
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/42)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.3 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.4 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=4
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762161, @last-run=1594762161, @exec-time=0, 
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/42, version=0.7.4)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n02 | action 3
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n01 | action 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'virsh_node2' not in cache
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_rsc_register) 	info: Cached agent information for 'virsh_node2'
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.4 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.5 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=5
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources:  <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-sta
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/14, version=0.7.5)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_monitor_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/43)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.5 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.6 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=6
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-sta
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/43, version=0.7.6)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.6 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.7 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=7
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:7;3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=9, @rc-code=7, @op-status=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/15, version=0.7.7)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=11, reply=1, notify=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:virsh_node2 action:monitor call_id:11
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:virsh_node2 action:monitor call_id:11  exit-code:7 exec-time:0ms queue-time:0ms
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 2 action 3 (virsh_node2_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=9
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok) | call=7 key=virsh_node1_stop_0 confirmed=true cib-update=44
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/44)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.7 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.8 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=8
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:0;1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=7, @rc-code=0, @op-status=0, @exec-time=1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/44, version=0.7.8)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of probe operation for virsh_node2 on el8-a01n01: 7 (not running) | call=11 key=virsh_node2_monitor_0 confirmed=true cib-update=45
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/45)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 2 action 1 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=7
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.8 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.9 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=9
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:7;2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=11, @rc-code=7, @op-status=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/45, version=0.7.9)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 2 action 2 (virsh_node2_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=11
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation virsh_node1_start_0 on el8-a01n02 | action 4
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n01 | action 6
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_start_0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=12, reply=1, notify=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:virsh_node2 action:start call_id:12
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.9 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.10 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=10
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @operation_key=virsh_node1_start_0, @operation=start, @transition-key=4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762161, @last-run=1594762161, @exec-time=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/16, version=0.7.10)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/46)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.10 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.7.11 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=11
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @operation_key=virsh_node2_start_0, @operation=start, @transition-key=6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/46, version=0.7.11)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.7.11 2
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.8.0 (null)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=8, @num_updates=0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <primitive class="systemd" id="hypervisor" type="libvirtd"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <operations>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="hypervisor-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="hypervisor-start-interval-0s" interval="0s" name="start" timeout="100"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="hypervisor-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </operations>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </primitive>
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.8.0)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 2 aborted by primitive.hypervisor 'create': Configuration change | cib=0.8.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.8.0
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-6.raw
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.8.0 of the CIB to disk (digest: 2c9ff6874f28590448a2f242c01625bf)
Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8PX8sO (digest: /var/lib/pacemaker/cib/cib.YoDg1d)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.8.0 2
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.9.0 (null)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: -- /cib/configuration/resources/primitive[@id='hypervisor']
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=9
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <clone id="hypervisor-clone"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <primitive class="systemd" id="hypervisor" type="libvirtd">
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <operations>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="hypervisor-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="hypervisor-start-interval-0s" interval="0s" name="start" timeout="100"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="hypervisor-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    </operations>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </primitive>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <meta_attributes id="hypervisor-clone-meta_attributes">
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="hypervisor-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="hypervisor-clone-meta_attributes-notify" name="notify" value="false"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </meta_attributes>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </clone>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.9.0)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by deletion of primitive[@id='hypervisor']: Configuration change | cib=0.9.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='hypervisor'] complete=false
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_device_remove) 	info: Device 'hypervisor' not found (2 active devices)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.9.0
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-7.raw
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.9.0 of the CIB to disk (digest: db64a16c52c066583a5b1a74240d556a)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AeCozH (digest: /var/lib/pacemaker/cib/cib.0ndsv8)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.9.0 2
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.10.0 (null)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=10
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <primitive class="systemd" id="drbd" type="drbd"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <operations>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="drbd-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="drbd-start-interval-0s" interval="0s" name="start" timeout="100"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <op id="drbd-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </operations>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </primitive>
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by primitive.drbd 'create': Configuration change | cib=0.10.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.10.0
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.10.0)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.10.0 of the CIB to disk (digest: 84477666f918034297fa4b47c59c0a6a)
Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uWgQLD (digest: /var/lib/pacemaker/cib/cib.6qhLc6)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.10.0 2
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.0 (null)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: -- /cib/configuration/resources/primitive[@id='drbd']
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @epoch=11
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/configuration/resources:  <clone id="drbd-clone"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <primitive class="systemd" id="drbd" type="drbd">
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <operations>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="drbd-monitor-interval-60" interval="60" name="monitor"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="drbd-start-interval-0s" interval="0s" name="start" timeout="100"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                      <op id="drbd-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    </operations>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </primitive>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  <meta_attributes id="drbd-clone-meta_attributes">
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="drbd-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                    <nvpair id="drbd-clone-meta_attributes-notify" name="notify" value="false"/>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                  </meta_attributes>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                </clone>
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by deletion of primitive[@id='drbd']: Configuration change | cib=0.11.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='drbd'] complete=false
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.11.0)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_device_remove) 	info: Device 'drbd' not found (2 active devices)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_cib_stonith_devices_v2) 	info: Updating device list from the cib: create resources
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (cib_devices_update) 	info: Updating devices to version 0.11.0
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_backup) 	info: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Wrote version 0.11.0 of the CIB to disk (digest: 2cddade26531e2c2dfcb3bac3401f84c)
Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_file_write_with_digest) 	info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.wPpOoC (digest: /var/lib/pacemaker/cib/cib.Lx5S85)
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:377569:stderr [ 2020-07-14 21:29:26,751 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:377569:stderr [  ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_1:377569:stderr [  ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377569] stderr: [ 2020-07-14 21:29:26,751 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377569] stderr: [  ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[377569] stderr: [  ]
Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (monitor). remaining timeout is 15
Jul 14 21:29:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: b7814e3f7afd5c57f438342d72812427 for 0.11.0 (0x555a63c3b420 0)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.0 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.1 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=10, @rc-code=1, @op-status=0, @exec-time=11419
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']:  <lrm_rsc_op id="virsh_node1_last_failure_0" operation_key="virsh_node1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" ca
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by operation virsh_node1_start_0 'modify' on el8-a01n02: Event failed | magic=0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.1 source=process_graph_event:453 complete=false
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_failcount) 	info: Updating failcount for virsh_node1 on el8-a01n02 after failed start: rc=1 (update=INFINITY, time=1594762173)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	notice: Transition 2 action 4 (virsh_node1_start_0 on el8-a01n02): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=10
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting fail-count-virsh_node1#start_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting last-failure-virsh_node1#start_0[el8-a01n02]: (unset) -> 1594762173 | from el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/17, version=0.11.1)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.1 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.2 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']:  <transient_attributes id="2"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                     <instance_attributes id="status-2">
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                       <nvpair id="status-2-fail-count-virsh_node1.start_0" name="fail-count-virsh_node1#start_0" value="INFINITY"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                     </instance_attributes>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                   </transient_attributes>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.11.2)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.11.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=false
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.2 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.3 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=3
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-last-failure-virsh_node1.start_0" name="last-failure-virsh_node1#start_0" value="1594762173"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by status-2-last-failure-virsh_node1.start_0 doing create last-failure-virsh_node1#start_0=1594762173: Transient attribute change | cib=0.11.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.11.3)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:378392:stderr [ 2020-07-14 21:29:32,948 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:378392:stderr [  ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_monitor_2:378392:stderr [  ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[378392] stderr: [ 2020-07-14 21:29:32,948 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[378392] stderr: [  ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[378392] stderr: [  ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (monitor) the maximum number of times (2) allowed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	notice: Operation 'monitor' [378392] for device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	info: finished - rsc:virsh_node2 action:start call_id:12  exit-code:1 exec-time:11463ms queue-time:0ms
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of start operation for virsh_node2 on el8-a01n01: 1 (error) | call=12 key=virsh_node2_start_0 confirmed=true cib-update=47
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/47)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.3 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.4 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=4
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=12, @rc-code=1, @op-status=0, @exec-time=11463
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']:  <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" ca
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/47, version=0.11.4)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by operation virsh_node2_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.4 source=process_graph_event:453 complete=false
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_failcount) 	info: Updating failcount for virsh_node2 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762173)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	notice: Transition 2 action 6 (virsh_node2_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=12
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting fail-count-virsh_node2#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting last-failure-virsh_node2#start_0[el8-a01n01]: (unset) -> 1594762173 | from el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 2 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.4 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.5 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=5
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/8, version=0.11.5)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.5 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.6 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=6
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762173"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/9, version=0.11.6)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by status-1-fail-count-virsh_node2.start_0 doing create fail-count-virsh_node2#start_0=INFINITY: Transient attribute change | cib=0.11.5 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 2 aborted by status-1-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762173: Transient attribute change | cib=0.11.6 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	 FAILED el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	 FAILED el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node2 on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for hypervisor:0 on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for hypervisor:1 on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for drbd:0 on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for drbd:1 on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       virsh_node1     (               el8-a01n02 )   due to node availability
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Recover    virsh_node2     (               el8-a01n01 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      hypervisor:0    (               el8-a01n02 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      hypervisor:1    (               el8-a01n01 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      drbd:0          (               el8-a01n02 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      drbd:1          (               el8-a01n01 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-3.bz2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_response) 	info: pe_calc calculation pe_calc-dc-1594762173-21 is obsolete
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	 FAILED el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	 FAILED el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node2 on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for hypervisor:0 on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for hypervisor:1 on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for drbd:0 on el8-a01n01
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for drbd:1 on el8-a01n02
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       virsh_node1     (               el8-a01n02 )   due to node availability
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Recover    virsh_node2     ( el8-a01n01 -> el8-a01n02 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      hypervisor:0    (               el8-a01n01 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      hypervisor:1    (               el8-a01n02 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      drbd:0          (               el8-a01n01 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Start      drbd:1          (               el8-a01n02 )  
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-4.bz2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 4 (ref=pe_calc-dc-1594762173-22) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation virsh_node1_stop_0 on el8-a01n02 | action 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n01 | action 1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_stop_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/51)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=13, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:virsh_node2 action:stop call_id:13
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	info: finished - rsc:virsh_node2 action:stop call_id:13  exit-code:0 exec-time:1ms queue-time:1ms
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.6 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.7 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=7
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173, @exec-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/51, version=0.11.7)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.7 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.8 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=8
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173, @exec-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/18, version=0.11.8)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.8 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.9 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=9
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']:  @transition-magic=0:0;2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=11, @rc-code=0, @op-status=0, @exec-time=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/19, version=0.11.9)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation hypervisor:0_monitor_0 locally on el8-a01n01 | action 3
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'hypervisor' not in cache
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'hypervisor:0' not in cache
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_rsc_register) 	info: Cached agent information for 'hypervisor'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_monitor_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=18, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:hypervisor action:monitor call_id:18
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation hypervisor:1_monitor_0 on el8-a01n02 | action 5
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation drbd:0_monitor_0 locally on el8-a01n01 | action 4
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'drbd' not in cache
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:18  exit-code:7 exec-time:8ms queue-time:0ms
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.9 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.10 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=10
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-statu
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/52, version=0.11.10)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.10 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.11 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=11
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources:  <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-statu
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/20, version=0.11.11)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_get_rsc_info) 	info: Agent information for 'drbd:0' not in cache
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_rsc_register) 	info: Cached agent information for 'drbd'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_monitor_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/53)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.11 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.12 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=12
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources:  <lrm_resource id="drbd" type="drbd" class="systemd"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-status="-1" inter
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/53, version=0.11.12)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=23, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:drbd action:monitor call_id:23
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.12 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.13 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=13
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:7;5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=16, @rc-code=7, @op-status=0, @exec-time=11
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/21, version=0.11.13)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation drbd:1_monitor_0 on el8-a01n02 | action 6
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 2 (virsh_node1_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=11
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 5 (hypervisor_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=16
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/54)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.13 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.14 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=14
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:0;1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=13, @rc-code=0, @op-status=0, @exec-time=1, @queue-time=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/54, version=0.11.14)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:23  exit-code:7 exec-time:11ms queue-time:1ms
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.14 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.15 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=15
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources:  <lrm_resource id="drbd" type="drbd" class="systemd"/>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                                <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-status="-1" inter
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++                                                              </lrm_resource>
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/22, version=0.11.15)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of stop operation for virsh_node2 on el8-a01n01: 0 (ok) | call=13 key=virsh_node2_stop_0 confirmed=true cib-update=54
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/55)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.15 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.16 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=16
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:7;3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=18, @rc-code=7, @op-status=0, @exec-time=8
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of probe operation for hypervisor on el8-a01n01: 7 (not running) | call=18 key=hypervisor_monitor_0 confirmed=true cib-update=55
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of probe operation for drbd on el8-a01n01: 7 (not running) | call=23 key=drbd_monitor_0 confirmed=true cib-update=56
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/55, version=0.11.16)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/56)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 1 (virsh_node2_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=13
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.16 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.17 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=17
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @transition-magic=0:7;4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=23, @rc-code=7, @op-status=0, @exec-time=11, @queue-time=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 3 (hypervisor_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=18
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation virsh_node2_start_0 on el8-a01n02 | action 7
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation hypervisor:0_start_0 locally on el8-a01n01 | action 9
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/56, version=0.11.17)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_start_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=24, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:hypervisor action:start call_id:24
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous start op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec_with_unit) 	debug: Calling StartUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation hypervisor:1_start_0 on el8-a01n02 | action 11
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.17 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.18 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=18
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @transition-magic=0:7;6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=21, @rc-code=7, @op-status=0, @exec-time=14, @queue-time=1
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 4 (drbd_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=23
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/23, version=0.11.18)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 6 (drbd_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=21
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation drbd:0_start_0 locally on el8-a01n01 | action 17
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_start_0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.18 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.19 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=19
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @operation_key=virsh_node2_start_0, @operation=start, @transition-key=7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=25, reply=1, notify=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:drbd action:start call_id:25
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous start op on systemd unit drbd named 'drbd'
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/24, version=0.11.19)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating start operation drbd:1_start_0 on el8-a01n02 | action 19
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.19 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.20 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=20
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @operation_key=hypervisor_start_0, @operation=start, @transition-key=11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/25, version=0.11.20)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.20 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.21 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=21
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @operation_key=drbd_start_0, @operation=start, @transition-key=19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @queue-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/26, version=0.11.21)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.21 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.22 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=22
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @operation_key=hypervisor_start_0, @operation=start, @transition-key=9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.11.22)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.22 2
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.23 (null)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=23
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @operation_key=drbd_start_0, @operation=start, @transition-key=17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @queue-time=0
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.11.23)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_exec_result) 	info: Call to start passed: /org/freedesktop/systemd1/job/4139
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: hypervisor start may still be in progress: re-scheduling (elapsed=246ms, remaining=99754ms, start_delay=2000ms)
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec_with_unit) 	debug: Calling StartUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_exec_result) 	info: Call to start passed: /org/freedesktop/systemd1/job/4223
Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: drbd start may still be in progress: re-scheduling (elapsed=506ms, remaining=99494ms, start_delay=2000ms)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.23 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.24 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=24
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:0;11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=23, @rc-code=0, @op-status=0, @exec-time=2222
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/27, version=0.11.24)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 11 (hypervisor_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=23
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation hypervisor:1_monitor_60000 on el8-a01n02 | action 12
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.24 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.25 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=25
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']:  <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/28, version=0.11.25)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.25 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.26 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=26
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']:  @transition-magic=0:0;12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=25, @rc-code=0, @op-status=0, @exec-time=3
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.11.26)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 12 (hypervisor_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=25
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:hypervisor action:monitor call_id:24
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: hypervisor start is now complete (elapsed=2255ms, remaining=97745ms): ok (0)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:24  exit-code:0 exec-time:2255ms queue-time:1ms
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of start operation for hypervisor on el8-a01n01: 0 (ok) | call=24 key=hypervisor_start_0 confirmed=true cib-update=59
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/59)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.26 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.27 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=27
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:0;9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=24, @rc-code=0, @op-status=0, @exec-time=2255, @queue-time=1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/59, version=0.11.27)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 9 (hypervisor_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=24
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation hypervisor:0_monitor_60000 locally on el8-a01n01 | action 10
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_monitor_60000
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/60)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.27 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.28 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=28
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']:  <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=26, reply=1, notify=0
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:hypervisor action:monitor call_id:26
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/60, version=0.11.28)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:26  exit-code:0 exec-time:4ms queue-time:0ms
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of monitor operation for hypervisor on el8-a01n01: 0 (ok) | call=26 key=hypervisor_monitor_60000 confirmed=false cib-update=61
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.28 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.29 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=29
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']:  @transition-magic=0:0;10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=26, @rc-code=0, @op-status=0, @exec-time=4
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.11.29)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 10 (hypervisor_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=26
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.29 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.30 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=30
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @transition-magic=0:0;19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=24, @rc-code=0, @op-status=0, @exec-time=2409, @queue-time=1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.11.30)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 19 (drbd_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=24
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation drbd:1_monitor_60000 on el8-a01n02 | action 20
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.30 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.31 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=31
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']:  <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.11.31)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.31 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.32 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=32
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']:  @transition-magic=0:0;20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=26, @rc-code=0, @op-status=0, @exec-time=3
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.11.32)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 20 (drbd_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=26
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:drbd action:monitor call_id:25
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: drbd start is now complete (elapsed=2511ms, remaining=97489ms): ok (0)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:25  exit-code:0 exec-time:2511ms queue-time:1ms
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of start operation for drbd on el8-a01n01: 0 (ok) | call=25 key=drbd_start_0 confirmed=true cib-update=62
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/62)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.32 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.33 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=33
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @transition-magic=0:0;17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=25, @rc-code=0, @op-status=0, @exec-time=2511, @queue-time=1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/62, version=0.11.33)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 17 (drbd_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=25
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating monitor operation drbd:0_monitor_60000 locally on el8-a01n01 | action 18
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_monitor_60000
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/63)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=27, reply=1, notify=0
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:drbd action:monitor call_id:27
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.33 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.34 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=34
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']:  <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/63, version=0.11.34)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:27  exit-code:0 exec-time:3ms queue-time:0ms
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of monitor operation for drbd on el8-a01n01: 0 (ok) | call=27 key=drbd_monitor_60000 confirmed=false cib-update=64
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/64)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.34 2
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.35 (null)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=35
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']:  @transition-magic=0:0;18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=27, @rc-code=0, @op-status=0, @exec-time=3
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/64, version=0.11.35)
Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 4 action 18 (drbd_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=27
Jul 14 21:29:40 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: 72c0a0507a5f4ea0970661199a542caa for 0.11.35 (0x555a63c3b420 0)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.35 2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.36 (null)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=36
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=22, @rc-code=1, @op-status=0, @exec-time=11492
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']:  <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" ca
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.11.36)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 4 aborted by operation virsh_node2_start_0 'modify' on el8-a01n02: Event failed | magic=0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.36 source=process_graph_event:453 complete=false
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_failcount) 	info: Updating failcount for virsh_node2 on el8-a01n02 after failed start: rc=1 (update=INFINITY, time=1594762184)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	notice: Transition 4 action 7 (virsh_node2_start_0 on el8-a01n02): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=22
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 4 (Complete=19, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting fail-count-virsh_node2#start_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting last-failure-virsh_node2#start_0[el8-a01n02]: (unset) -> 1594762184 | from el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.36 2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.37 (null)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=37
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/10, version=0.11.37)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.37 2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.38 (null)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=38
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762184"/>
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/11, version=0.11.38)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 4 aborted by status-2-fail-count-virsh_node2.start_0 doing create fail-count-virsh_node2#start_0=INFINITY: Transient attribute change | cib=0.11.37 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 4 aborted by status-2-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762184: Transient attribute change | cib=0.11.38 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	 FAILED el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) 	info:  Start recurring monitor (60s) for virsh_node2 on el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Recover    virsh_node2     (               el8-a01n02 )  
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Started el8-a01n01)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Started el8-a01n02)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   drbd:0	(Started el8-a01n01)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   drbd:1	(Started el8-a01n02)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-5.bz2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_response) 	info: pe_calc calculation pe_calc-dc-1594762184-38 is obsolete
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n02 is active
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is online
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	 FAILED el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       virsh_node2     (               el8-a01n02 )   due to node availability
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Started el8-a01n01)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Started el8-a01n02)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   drbd:0	(Started el8-a01n01)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   drbd:1	(Started el8-a01n02)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-6.bz2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 6 (ref=pe_calc-dc-1594762184-39) derived from /var/lib/pacemaker/pengine/pe-input-6.bz2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n02 | action 3
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.38 2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.39 (null)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=39
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762184, @last-run=1594762184, @exec-time=0
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.11.39)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.39 2
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.40 (null)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=40
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']:  @transition-magic=0:0;3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=27, @rc-code=0, @op-status=0, @exec-time=1
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/35, version=0.11.40)
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 6 action 3 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=27
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 6 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-6.bz2): Complete
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_log) 	info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:29:49 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: 8251df6e54b59aa0e832ca3aef06d2cb for 0.11.40 (0x555a63c3b420 0)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_expected) 	info: handle_request: Node el8-a01n02[2] - expected state is now down (was member)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_shutdown_request) 	info: Creating shutdown request for el8-a01n02 (state=S_IDLE)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting shutdown[el8-a01n02]: (unset) -> 1594762228 | from el8-a01n01
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.40 2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.41 (null)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=41
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-shutdown" name="shutdown" value="1594762228"/>
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/12, version=0.11.41)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 6 aborted by status-2-shutdown doing create shutdown=1594762228: Transient attribute change | cib=0.11.41 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) 	info: Node el8-a01n01 is active
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is online
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n02
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n02
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Started el8-a01n01)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       hypervisor:1    (               el8-a01n02 )   due to node availability
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   drbd:0	(Started el8-a01n01)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	notice: Calculated transition 7, saving inputs in /var/lib/pacemaker/pengine/pe-input-7.bz2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 7 (ref=pe_calc-dc-1594762228-41) derived from /var/lib/pacemaker/pengine/pe-input-7.bz2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation hypervisor_stop_0 on el8-a01n02 | action 7
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 14
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.41 2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.42 (null)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=42
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762228, @last-run=1594762228, @exec-time=0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/36, version=0.11.42)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.42 2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.43 (null)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=43
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @operation_key=drbd_stop_0, @operation=stop, @transition-key=14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762228, @last-run=1594762228, @exec-time=0, @queue-time=0
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/37, version=0.11.43)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd          [375401] (crm_signal_dispatch) 	notice: Caught 'Terminated' signal | 15 (invoking handler)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd          [375401] (pcmk_shutdown_worker) 	notice: Shutting down Pacemaker
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd          [375401] (stop_child) 	notice: Stopping pacemaker-controld | sent signal 15 to process 375414
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_signal_dispatch) 	notice: Caught 'Terminated' signal | 15 (invoking handler)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_shutdown) 	notice: Shutting down cluster resource manager | limit=1200000ms
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_log) 	warning: Input I_SHUTDOWN received in state S_TRANSITION_ENGINE from crm_shutdown
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_shutdown_req) 	info: Sending shutdown request to all peers (DC is el8-a01n01)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (crm_update_peer_expected) 	info: handle_request: Node el8-a01n01[1] - expected state is now down (was member)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_shutdown_request) 	info: Creating shutdown request for el8-a01n01 (state=S_POLICY_ENGINE)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting shutdown[el8-a01n01]: (unset) -> 1594762228 | from el8-a01n01
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.43 2
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.44 (null)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=44
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-shutdown" name="shutdown" value="1594762228"/>
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/13, version=0.11.44)
Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 7 aborted by status-1-shutdown doing create shutdown=1594762228: Transient attribute change | cib=0.11.44 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.44 2
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.45 (null)
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=45
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:0;7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2224
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/38, version=0.11.45)
Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 7 action 7 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=29
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (recurring_action_timer) 	debug: Scheduling another invocation of hypervisor_status_60000
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:26  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: afb079b332593e47dff6eff041966396 for 0.11.45 (0x555a63c3b420 0)
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (recurring_action_timer) 	debug: Scheduling another invocation of drbd_status_60000
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:27  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (recurring_action_timer) 	debug: Scheduling another invocation of hypervisor_status_60000
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:26  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (recurring_action_timer) 	debug: Scheduling another invocation of drbd_status_60000
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:27  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.45 2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.46 (null)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=46
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']:  @transition-magic=2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=31, @rc-code=198, @op-status=2, @exec-time=99991
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']:  <lrm_rsc_op id="drbd_last_failure_0" operation_key="drbd_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="31" rc-code=
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/39, version=0.11.46)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 7 aborted by operation drbd_stop_0 'modify' on el8-a01n02: Event failed | magic=2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.46 source=process_graph_event:453 complete=false
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (update_failcount) 	info: Updating failcount for drbd on el8-a01n02 after failed stop: rc=198 (update=INFINITY, time=1594762328)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-attrd     [375412] (attrd_peer_update) 	notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594762328 | from el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	notice: Transition 7 action 14 (drbd_stop_0 on el8-a01n02): expected 'ok' but got 'OCF_TIMEOUT' | target-rc=0 rc=198 call-id=31
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 7 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-7.bz2): Complete
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.46 2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.47 (null)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=47
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-fail-count-drbd.stop_0" name="fail-count-drbd#stop_0" value="INFINITY"/>
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/14, version=0.11.47)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 7 aborted by status-2-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change | cib=0.11.47 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.47 2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.48 (null)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=48
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']:  <nvpair id="status-2-last-failure-drbd.stop_0" name="last-failure-drbd#stop_0" value="1594762328"/>
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/15, version=0.11.48)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	info: Transition 7 aborted by status-2-last-failure-drbd.stop_0 doing create last-failure-drbd#stop_0=1594762328: Transient attribute change | cib=0.11.48 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n02 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       hypervisor:0    (               el8-a01n01 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 8 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-0.bz2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (handle_response) 	info: pe_calc calculation pe_calc-dc-1594762328-45 is obsolete
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n02 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       hypervisor:0    (               el8-a01n01 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 9 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-0.bz2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 9 (ref=pe_calc-dc-1594762328-46) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=4 timeout=60000
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_rsc_command) 	notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n01 | action 5
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=2bbcc655-c155-4908-a792-c918b427a38e state=0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (cancel_recurring_action) 	info: Cancelling systemd operation hypervisor_status_60000
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:26  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_cancel operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 2bbcc655-c155-4908-a792-c918b427a38e
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 2bbcc655-c155-4908-a792-c918b427a38e
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_lrm_rsc_op) 	info: Performing key=5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_stop_0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=2bbcc655-c155-4908-a792-c918b427a38e
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/72)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (process_lrmd_message) 	debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=29, reply=1, notify=0
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	info: executing - rsc:hypervisor action:stop call_id:29
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous stop op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.48 2
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.49 (null)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=49
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762328, @last-run=1594762328, @exec-time=0, @qu
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/72, version=0.11.49)
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	info: Result of monitor operation for hypervisor on el8-a01n01: Cancelled | call=26 key=hypervisor_monitor_60000 confirmed=true
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec_with_unit) 	debug: Calling StopUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_exec_result) 	info: Call to stop passed: /org/freedesktop/systemd1/job/4328
Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: hypervisor stop may still be in progress: re-scheduling (elapsed=267ms, remaining=99733ms, start_delay=2000ms)
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_execute) 	debug: executing - rsc:hypervisor action:monitor call_id:29
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (action_complete) 	debug: hypervisor stop is now complete (elapsed=2271ms, remaining=97729ms): not running (7)
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:hypervisor action:monitor call_id:29  exit-code:0 exec-time:2271ms queue-time:0ms
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Forwarding cib_modify operation for section status to all (origin=local/crmd/73)
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_lrm_event) 	notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok) | call=29 key=hypervisor_stop_0 confirmed=true cib-update=73
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: --- 0.11.49 2
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: Diff: +++ 0.11.50 (null)
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib:  @num_updates=50
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_perform_op) 	info: +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']:  @transition-magic=0:0;5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2271
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_request) 	info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/73, version=0.11.50)
Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-controld  [375414] (process_graph_event) 	info: Transition 9 action 5 (hypervisor_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=29
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:388864:stderr [ 2020-07-14 21:32:13,484 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:388864:stderr [  ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:388864:stderr [  ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[388864] stderr: [ 2020-07-14 21:32:13,484 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[388864] stderr: [  ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[388864] stderr: [  ]
Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
Jul 14 21:32:15 el8-a01n01.digimer.ca pacemaker-based     [375409] (cib_process_ping) 	info: Reporting our current digest to el8-a01n01: 55204ae50dead0b498c548c5d466607c for 0.11.50 (0x555a63c3b420 0)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:389235:stderr [ 2020-07-14 21:32:19,741 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:389235:stderr [  ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:389235:stderr [  ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389235] stderr: [ 2020-07-14 21:32:19,741 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389235] stderr: [  ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389235] stderr: [  ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [389235] (call 2 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.2bbcc655: No route to host
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 2/4:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 2 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 9 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=2bbcc655-c155-4908-a792-c918b427a38e
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 9 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Complete
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 10 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 10 (ref=pe_calc-dc-1594762339-48) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=75793d25-0284-4916-823f-ebadcf09771b state=0
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 75793d25-0284-4916-823f-ebadcf09771b
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 75793d25-0284-4916-823f-ebadcf09771b
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=75793d25-0284-4916-823f-ebadcf09771b
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:389538:stderr [ 2020-07-14 21:32:24,971 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:389538:stderr [  ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:389538:stderr [  ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389538] stderr: [ 2020-07-14 21:32:24,971 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389538] stderr: [  ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[389538] stderr: [  ]
Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390256:stderr [ 2020-07-14 21:32:31,205 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390256:stderr [  ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390256:stderr [  ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390256] stderr: [ 2020-07-14 21:32:31,205 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390256] stderr: [  ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390256] stderr: [  ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [390256] (call 3 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.75793d25: No route to host
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 3/3:10:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 3 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 10 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=75793d25-0284-4916-823f-ebadcf09771b
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 10 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 11 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 11 (ref=pe_calc-dc-1594762351-49) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=5be1f6ba-66c0-4d35-920f-a083bf59809c state=0
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 5be1f6ba-66c0-4d35-920f-a083bf59809c
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 5be1f6ba-66c0-4d35-920f-a083bf59809c
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=5be1f6ba-66c0-4d35-920f-a083bf59809c
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (recurring_action_timer) 	debug: Scheduling another invocation of drbd_status_60000
Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (systemd_unit_exec) 	debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (pcmk_dbus_get_property) 	debug: Calling: GetAll on org.freedesktop.systemd1
Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd     [375411] (log_finished) 	debug: finished - rsc:drbd action:monitor call_id:27  exit-code:0 exec-time:0ms queue-time:0ms
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:390569:stderr [ 2020-07-14 21:32:36,434 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:390569:stderr [  ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:390569:stderr [  ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390569] stderr: [ 2020-07-14 21:32:36,434 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390569] stderr: [  ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390569] stderr: [  ]
Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390879:stderr [ 2020-07-14 21:32:42,776 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390879:stderr [  ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:390879:stderr [  ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390879] stderr: [ 2020-07-14 21:32:42,776 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390879] stderr: [  ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[390879] stderr: [  ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [390879] (call 4 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.5be1f6ba: No route to host
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 4/3:11:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 4 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 11 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=5be1f6ba-66c0-4d35-920f-a083bf59809c
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 11 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 12 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 12 (ref=pe_calc-dc-1594762362-50) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=9fc8382b-26df-4e45-8f59-affe54245144 state=0
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 9fc8382b-26df-4e45-8f59-affe54245144
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 9fc8382b-26df-4e45-8f59-affe54245144
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=9fc8382b-26df-4e45-8f59-affe54245144
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391149:stderr [ 2020-07-14 21:32:48,173 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391149:stderr [  ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391149:stderr [  ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391149] stderr: [ 2020-07-14 21:32:48,173 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391149] stderr: [  ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391149] stderr: [  ]
Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:391399:stderr [ 2020-07-14 21:32:54,383 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:391399:stderr [  ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:391399:stderr [  ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391399] stderr: [ 2020-07-14 21:32:54,383 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391399] stderr: [  ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391399] stderr: [  ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [391399] (call 5 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.9fc8382b: No route to host
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 5/3:12:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 5 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 12 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=9fc8382b-26df-4e45-8f59-affe54245144
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 12 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 13 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 13 (ref=pe_calc-dc-1594762374-51) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=d3fc9798-6df8-485b-a7d3-e904d314cf81 state=0
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) d3fc9798-6df8-485b-a7d3-e904d314cf81
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) d3fc9798-6df8-485b-a7d3-e904d314cf81
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=d3fc9798-6df8-485b-a7d3-e904d314cf81
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391738:stderr [ 2020-07-14 21:32:59,589 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391738:stderr [  ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:391738:stderr [  ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391738] stderr: [ 2020-07-14 21:32:59,589 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391738] stderr: [  ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[391738] stderr: [  ]
Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392102:stderr [ 2020-07-14 21:33:05,784 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392102:stderr [  ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392102:stderr [  ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392102] stderr: [ 2020-07-14 21:33:05,784 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392102] stderr: [  ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392102] stderr: [  ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [392102] (call 6 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.d3fc9798: No route to host
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 6/3:13:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 6 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 13 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=d3fc9798-6df8-485b-a7d3-e904d314cf81
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 13 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 14 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 14 (ref=pe_calc-dc-1594762385-52) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=3e12455e-82d1-4460-90db-9dcb37d6de0e state=0
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 3e12455e-82d1-4460-90db-9dcb37d6de0e
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 3e12455e-82d1-4460-90db-9dcb37d6de0e
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=3e12455e-82d1-4460-90db-9dcb37d6de0e
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:392399:stderr [ 2020-07-14 21:33:11,068 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:392399:stderr [  ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_1:392399:stderr [  ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392399] stderr: [ 2020-07-14 21:33:11,068 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392399] stderr: [  ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392399] stderr: [  ]
Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (internal_stonith_action_execute) 	info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392682:stderr [ 2020-07-14 21:33:17,444 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392682:stderr [  ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (operation_finished) 	notice: fence_virsh_reboot_2:392682:stderr [  ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392682] stderr: [ 2020-07-14 21:33:17,444 ERROR: Unable to connect/login to fencing device ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392682] stderr: [  ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_action) 	warning: fence_virsh[392682] stderr: [  ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (update_remaining_timeout) 	info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (log_operation) 	error: Operation 'reboot' [392682] (call 7 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_choose_peer) 	notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (remote_op_done) 	error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.3e12455e: No route to host
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 7/3:14:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_callback) 	notice: Stonith operation 7 for el8-a01n02 failed (No route to host): aborting transition.
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (abort_transition_graph) 	notice: Transition 14 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (tengine_stonith_notify) 	notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=3e12455e-82d1-4460-90db-9dcb37d6de0e
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (run_graph) 	notice: Transition 14 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) 	info: Input has not changed since last time, not saving to disk
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n01 is shutting down
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) 	info: Node el8-a01n02 is shutting down
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) 	warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) 	warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 1 is already processed
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) 	info: Node 2 is already processed
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node1	(stonith:fence_virsh):	Stopped
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info: virsh_node2	(stonith:fence_virsh):	Stopped
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: hypervisor-clone [hypervisor]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Stopped: [ el8-a01n01 el8-a01n02 ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) 	info:  Clone Set: drbd-clone [drbd]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) 	info:      drbd	(systemd:drbd):	 FAILED el8-a01n02
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) 	info:      Started: [ el8-a01n01 ]
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n01
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n01
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node1 has failed INFINITY times on el8-a01n02
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: virsh_node2 has failed INFINITY times on el8-a01n02
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:0 has failed INFINITY times on el8-a01n02
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) 	info: drbd:1 has failed INFINITY times on el8-a01n02
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) 	warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node1 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource virsh_node2 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:0 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource hypervisor:1 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:0 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) 	info: Resource drbd:1 cannot run anywhere
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) 	notice: Scheduling shutdown of node el8-a01n01
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) 	warning: Scheduling Node el8-a01n02 for STONITH
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) 	notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Shutdown el8-a01n01
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) 	notice:  * Fence (reboot) el8-a01n02 'drbd:1 failed there'
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node1	(Stopped)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   virsh_node2	(Stopped)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:0	(Stopped)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) 	info: Leave   hypervisor:1	(Stopped)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:0          (               el8-a01n01 )   due to node availability
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) 	notice:  * Stop       drbd:1          (               el8-a01n02 )   due to node availability
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) 	warning: Calculated transition 15 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_state_transition) 	info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (do_te_invoke) 	info: Processing graph 15 (ref=pe_calc-dc-1594762397-53) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld  [375414] (te_fence_node) 	notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (handle_request) 	notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (initiate_remote_stonith_op) 	notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=97afe994-2cc3-45d0-887f-c94bed0d27e1 state=0
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 97afe994-2cc3-45d0-887f-c94bed0d27e1
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 97afe994-2cc3-45d0-887f-c94bed0d27e1
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (process_remote_stonith_query) 	info: All query replies have arrived, continuing (2 expected/2 received) 
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=97afe994-2cc3-45d0-887f-c94bed0d27e1
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (call_remote_stonith) 	notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (can_fence_host_with_device) 	notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced    [375410] (stonith_fence_get_devices_cb) 	info: Found 1 matching devices for 'el8-a01n02'