[root@el8-a01n01 ~]# cat /var/log/pacemaker/el8-a01n01/pacemaker.log.drbd-down-up-to-crash Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (get_cluster_type) info: Detected an active 'corosync' cluster Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (mcp_read_config) info: Reading configure for stack: corosync Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (main) notice: Starting Pacemaker 2.0.3-5.el8_2.1 | build=4b1f869f0f features: generated-manpages agent-manpages ncurses libqb-logging libqb-ipc systemd nagios corosync-native atomic-attrd acls Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (main) info: Maximum core file size is: 18446744073709551615 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (qb_ipcs_us_publish) info: server name: pacemakerd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to lrmd IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to cib_ro IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to crmd IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to pengine IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk__ipc_is_authentic_process_active) info: Could not connect to stonith-ng IPC: Connection refused Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Created entry 0dbc95c7-f89b-471a-8607-0f48f89e06ac/0x564a746618c0 for node el8-a01n01/1 (1 total) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (cluster_connect_quorum) warning: Quorum lost Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Created entry ac6b79a3-392f-40d9-833e-d3bceee892f4/0x564a74663860 for node el8-a01n02/2 (2 total) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Using uid=189 and group=189 for process pacemaker-based Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32267 for process pacemaker-based Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32268 for process pacemaker-fenced Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32269 for process pacemaker-execd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Using uid=189 and group=189 for process pacemaker-attrd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32270 for process pacemaker-attrd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Using uid=189 and group=189 for process pacemaker-schedulerd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32271 for process pacemaker-schedulerd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Using uid=189 and group=189 for process pacemaker-controld Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (start_child) info: Forked child 32273 for process pacemaker-controld Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (main) notice: Pacemaker daemon successfully started and accepting connections Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_quorum_notification) info: Quorum still lost | membership=171 members=1 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (crm_reap_unseen_nodes) info: State of node el8-a01n02[2] is still unknown Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 0: node 1 pid 32266 joined via cpg_join Jul 14 22:08:15 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 0: el8-a01n01 (node 1 pid 32266) is member Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-execd [32269] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-execd [32269] (main) notice: Starting Pacemaker local executor Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_ipcs_us_publish) info: server name: lrmd Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-execd [32269] (main) notice: Pacemaker local executor successfully started and accepting connections Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (main) notice: Starting Pacemaker fencer Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (main) notice: Starting Pacemaker CIB manager Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Created entry 7351d875-2fce-4558-86b8-d280d8bde902/0x555b7d3c4d60 for node el8-a01n01/1 (1 total) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (retrieveCib) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-attrd [32270] (main) notice: Starting Pacemaker node attribute manager Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-controld [32273] (main) notice: Starting Pacemaker controller Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) info: Input I_STARTUP received in state S_STARTING from crmd_init Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (main) notice: Starting Pacemaker scheduler Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (qb_ipcs_us_publish) info: server name: pengine Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (main) notice: Pacemaker scheduler successfully started and accepting connections Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-fenced [32268] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Created entry f226808d-96b6-4acd-93c2-315848293505/0x55db3e137400 for node el8-a01n01/1 (1 total) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (qb_ipcs_us_publish) info: server name: cib_ro Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (qb_ipcs_us_publish) info: server name: cib_rw Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (qb_ipcs_us_publish) info: server name: cib_shm Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (main) notice: Pacemaker CIB manager successfully started and accepting connections Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 0: node 1 pid 32267 joined via cpg_join Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 0: el8-a01n01 (node 1 pid 32267) is member Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-17.raw Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Wrote version 0.13.0 of the CIB to disk (digest: 721386e5982712a0d17a05527f015cab) Jul 14 22:08:15 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.UA6fqz (digest: /var/lib/pacemaker/cib/cib.PHKShp) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 1: node 2 pid 1814 joined via cluster join Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n01 (node 1 pid 32266) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n02 (node 2 pid 1814) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining, last=7): Try again (6) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_quorum_notification) notice: Quorum acquired | membership=175 members=2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 1: node 2 pid 1815 joined via cluster join Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n01 (node 1 pid 32267) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Created entry d77f39fc-5606-44c4-9222-a71b2ecc7003/0x55db3e13ca40 for node el8-a01n02/2 (2 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n02 (node 2 pid 1815) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n02/attrd/2, version=0.13.0) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/4, version=0.13.0) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (main) info: CIB connection active Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Created entry 2b8ecfae-8c19-437c-a944-009ce327fd71/0x563312e87eb0 for node el8-a01n01/1 (1 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Cluster node el8-a01n01 is now in unknown state Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Created entry 1327755a-a2a5-437e-a2aa-673f834532ac/0x5650d843ea20 for node el8-a01n01/1 (1 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Node el8-a01n01 is now a peer | DC= old=0x0000000 new=0x4000000 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (setup_cib) info: Watching for stonith topology changes Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (qb_ipcs_us_publish) info: server name: stonith-ng Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (main) notice: Pacemaker fencer successfully started and accepting connections Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 0: node 1 pid 32268 joined via cpg_join Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 0: el8-a01n01 (node 1 pid 32268) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (init_cib_cache_cb) info: Updating device list from the cib: init Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (cib_devices_update) info: Updating devices to version 0.13.0 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (main) info: Cluster connection active Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_erase_attrs) info: Clearing transient attributes from CIB | xpath=//node_state[@uname='el8-a01n01']/transient_attributes Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes to all (origin=local/attrd/2) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (cluster_connect_quorum) notice: Quorum acquired Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes: OK (rc=0, origin=el8-a01n01/attrd/2, version=0.13.0) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (qb_ipcs_us_publish) info: server name: attrd Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (main) notice: Pacemaker node attribute manager successfully started and accepting connections Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (pcmk_cpg_membership) info: Group attrd event 0: node 1 pid 32270 joined via cpg_join Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n01 (node 1 pid 32270) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Created entry 8c5373b7-f283-4dbc-a37d-cd18f862b0ca/0x563312e8a790 for node el8-a01n02/2 (2 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Cluster node el8-a01n02 is now in unknown state Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Created entry 0852aa7d-2066-4b25-be7f-05aacf7d0bf2/0x5650d8445060 for node el8-a01n02/2 (2 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n02 (node 2 pid 1818) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_check_for_new_writer) notice: Recorded new attribute writer: el8-a01n02 (was unset) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_message) info: Processing sync-response from el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2 | from el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2 | from el8-a01n01 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_ha_control) info: Connected to the cluster Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/4) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (lrmd_ipc_connect) info: Connecting to executor Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (crm_client_new) debug: Connecting 0x55c354d31fb0 for uid=189 gid=189 pid=32273 id=ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (handle_new_connection) debug: IPC credentials authenticated (/dev/shm/qb-32269-32273-9-agJx1C/qb) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_ipcs_shm_connect) debug: connecting to client [32273] Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/4, version=0.13.0) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed register operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_control) info: Connection to the executor established Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_started) info: Delaying start, no membership data (0000000000100000) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_started) info: Delaying start, no membership data (0000000000100000) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_quorum_notification) info: Quorum retained | membership=175 members=2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Cluster node el8-a01n01 is now member (was in unknown state) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Cluster node el8-a01n02 is now member (was in unknown state) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_started) info: Delaying start, Config not read (0000000000000040) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (qb_ipcs_us_publish) info: server name: crmd Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_started) notice: Pacemaker controller successfully started and accepting connections Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) info: Input I_PENDING received in state S_STARTING from do_started Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 32334 exited with rc=0 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (stonith_device_register) notice: Added 'virsh_node1' to the device list (1 active devices) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (stonith_device_register) notice: Added 'virsh_node2' to the device list (2 active devices) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 2 pid 1816 joined via cluster join Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n01 (node 1 pid 32268) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Created entry 5c118c0e-539c-4de8-a7b4-b7b7d2871e87/0x555b7d4e6d80 for node el8-a01n02/2 (2 total) Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n02 (node 2 pid 1816) is member Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 22:08:16 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_cpg_membership) info: Group crmd event 0: node 1 pid 32273 joined via cpg_join Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n01 (node 1 pid 32273) is member Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n02 (node 2 pid 1820) is member Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Node el8-a01n02 is now a peer | DC= old=0x0000000 new=0x4000000 Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 30 seconds Jul 14 22:08:17 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_connect_stonith) notice: Fencer successfully connected Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (election_count_vote) info: election-DC round 1 (owner node ID 2) pass: vote from el8-a01n02 (Uptime) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_PENDING -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (election_check) info: election-DC won by local node Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_control) info: Registering TE UUID: ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (set_graph_functions) info: Setting custom graph functions Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_dc_takeover) info: Taking over DC status for this partition Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_readwrite) info: We are now in R/W mode Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/8) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/10) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/8, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/10, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/12) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (join_make_offer) info: Making join-1 offers based on membership event 175 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (join_make_offer) info: Sending join-1 offer to el8-a01n02 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (join_make_offer) info: Sending join-1 offer to el8-a01n01 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_dc_join_offer_all) info: Waiting on join-1 requests from 2 outstanding nodes Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/12, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/14) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (update_dc) info: Set DC to el8-a01n01 (3.3.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_expected) info: update_dc: Node el8-a01n01[1] - expected state is now member (was (null)) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/14, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_expected) info: do_dc_join_filter_offer: Node el8-a01n02[2] - expected state is now member (was (null)) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_replace) info: Digest matched on replace from el8-a01n01: e66cd993ae6725062fe95afbc6bd62d5 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_replace) info: Replaced 0.13.0 with 0.13.0 from el8-a01n01 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (controld_delete_node_state) info: Deleting resource history for node el8-a01n01 (via CIB call 20) | xpath=//node_state[@uname='el8-a01n01']/lrm Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (controld_delete_node_state) info: Deleting resource history for node el8-a01n02 (via CIB call 22) | xpath=//node_state[@uname='el8-a01n02']/lrm Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=el8-a01n01/crmd/17, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/18) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/19) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm to all (origin=local/crmd/20) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/21) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm to all (origin=local/crmd/22) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/18, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/19, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm: OK (rc=0, origin=el8-a01n01/crmd/20, version=0.13.0) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.0 2 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.1 (null) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=1 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status: Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/21, version=0.13.1) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm: OK (rc=0, origin=el8-a01n01/crmd/22, version=0.13.1) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/23) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/26) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.1 2 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.2 (null) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=2 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status: Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/23, version=0.13.2) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/26, version=0.13.2) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.2 2 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.3 (null) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=3 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition, @join=member, @expected=member Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition, @join=member, @expected=member Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/27, version=0.13.3) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/28) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.3 2 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.4 (null) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=4, @dc-uuid=1 Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/28, version=0.13.4) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-18.raw Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Wrote version 0.13.0 of the CIB to disk (digest: 96d8141ab40cd17f4dfe3f1be9e92394) Jul 14 22:08:37 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.n4El8S (digest: /var/lib/pacemaker/cib/cib.EadY4L) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is online Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is online Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ] Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ] Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:0 on el8-a01n01 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:1 on el8-a01n02 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n02 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start virsh_node1 ( el8-a01n01 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start virsh_node2 ( el8-a01n02 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start hypervisor:0 ( el8-a01n01 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start hypervisor:1 ( el8-a01n02 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start drbd:0 ( el8-a01n01 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start drbd:1 ( el8-a01n02 ) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-14.bz2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_invoke) info: Processing graph 0 (ref=pe_calc-dc-1594764518-9) derived from /var/lib/pacemaker/pengine/pe-input-14.bz2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n02 | action 5 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n01 | action 1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node1' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node1' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=1:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node1_monitor_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.4 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.5 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=5 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/9, version=0.13.5) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.5 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.6 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=6 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;5:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=5, @rc-code=7, @op-status=0, @exec-time=8 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/10, version=0.13.6) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 33668 exited with rc=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=5, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:5 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (stonith_api_signon) debug: Attempting fencer connection by pacemaker-execd with mainloop Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n02 | action 6 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.6 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.7 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=7 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n01 | action 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/31, version=0.13.7) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.7 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.8 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=8 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/11, version=0.13.8) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (stonith_api_signon) debug: Connection to fencer by pacemaker-execd succeeded (registration token: 27f0072c-422a-429c-b1a5-c89ea7ca3414) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.8 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.9 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=9 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;6:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=9, @rc-code=7, @op-status=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/12, version=0.13.9) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:5 exit-code:7 exec-time:8ms queue-time:1ms Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node2' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node2' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=2:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node2_monitor_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/32) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=9, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:9 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:9 exit-code:7 exec-time:0ms queue-time:1ms Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.9 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.10 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=10 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 5 (virsh_node1_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=5 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/32, version=0.13.10) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 6 (virsh_node2_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=9 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of probe operation for virsh_node1 on el8-a01n01: 7 (not running) | call=5 key=virsh_node1_monitor_0 confirmed=true cib-update=33 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/33) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.10 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.11 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=11 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;1:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=5, @rc-code=7, @op-status=0, @exec-time=8, @queue-time=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/33, version=0.13.11) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of probe operation for virsh_node2 on el8-a01n01: 7 (not running) | call=9 key=virsh_node2_monitor_0 confirmed=true cib-update=34 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/34) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.11 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.12 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=12 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;2:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=9, @rc-code=7, @op-status=0, @queue-time=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 1 (virsh_node1_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=5 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/34, version=0.13.12) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 2 (virsh_node2_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=9 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 locally on el8-a01n01 | action 9 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=9:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node1_start_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/35) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.12 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.13 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=13 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=9:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;9:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @queue-time=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/35, version=0.13.13) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=10, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:virsh_node1 action:start call_id:10 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 on el8-a01n02 | action 11 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_0 locally on el8-a01n01 | action 3 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.13 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.14 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=14 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=11:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;11:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/13, version=0.13.14) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor:0' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_rsc_register) info: Cached agent information for 'hypervisor' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=3:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=hypervisor_monitor_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.14 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.15 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=15 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/36, version=0.13.15) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=15, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:15 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_0 on el8-a01n02 | action 7 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.15 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.16 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=16 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/14, version=0.13.16) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.16 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.17 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=17 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;7:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=15, @rc-code=7, @op-status=0, @exec-time=35 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/15, version=0.13.17) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 7 (hypervisor_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=15 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_0 on el8-a01n02 | action 8 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:15 exit-code:7 exec-time:45ms queue-time:0ms Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of probe operation for hypervisor on el8-a01n01: 7 (not running) | call=15 key=hypervisor_monitor_0 confirmed=true cib-update=37 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/37) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.17 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.18 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=18 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;3:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=15, @rc-code=7, @op-status=0, @exec-time=45 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/37, version=0.13.18) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 3 (hypervisor_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=15 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_0 locally on el8-a01n01 | action 4 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'drbd' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.18 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.19 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=19 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/16, version=0.13.19) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_get_rsc_info) info: Agent information for 'drbd:0' not in cache Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_rsc_register) info: Cached agent information for 'drbd' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=4:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=drbd_monitor_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/38) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=20, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:20 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.19 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.20 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=20 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/38, version=0.13.20) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.20 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.21 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=21 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;8:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=20, @rc-code=7, @op-status=0, @exec-time=11 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/17, version=0.13.21) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 8 (drbd_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=20 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation hypervisor:1_start_0 on el8-a01n02 | action 15 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:20 exit-code:7 exec-time:11ms queue-time:0ms Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of probe operation for drbd on el8-a01n01: 7 (not running) | call=20 key=drbd_monitor_0 confirmed=true cib-update=39 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.21 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.22 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=22 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=15:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;15:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/18, version=0.13.22) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/39) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.22 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.23 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=23 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;4:0:7:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=20, @rc-code=7, @op-status=0, @exec-time=11 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/39, version=0.13.23) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 4 (drbd_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=20 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation hypervisor:0_start_0 locally on el8-a01n01 | action 13 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=13:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=hypervisor_start_0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/40) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=21, reply=1, notify=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:hypervisor action:start call_id:21 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit libvirtd named 'hypervisor' Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.23 2 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.24 (null) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=24 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=13:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;13:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/40, version=0.13.24) Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec_with_unit) debug: Calling StartUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/1743 Jul 14 22:08:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: hypervisor start may still be in progress: re-scheduling (elapsed=241ms, remaining=99759ms, start_delay=2000ms) Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.24 2 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.25 (null) Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=25 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;11:0:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=10, @rc-code=0, @op-status=0, @exec-time=809 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/19, version=0.13.25) Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 0 action 11 (virsh_node2_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=10 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_60000 on el8-a01n02 | action 12 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.25 2 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.13.26 (null) Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=26 Jul 14 22:08:39 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 22:08:46 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_procfs_pid_of) info: Found pacemaker-based active as process 32267 Jul 14 22:08:46 el8-a01n01.digimer.ca pacemaker-controld [32273] (throttle_send_command) info: New throttle mode: negligible load (was undetermined) Jul 14 22:08:48 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 0a257520f27bc31c4e11ed12cf711cab for 0.13.44 (0x55db3dd56bd0 0) Jul 14 22:09:40 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:09:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:09:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:09:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:09:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:09:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:600ms queue-time:0ms Jul 14 22:09:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 22:09:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:09:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:09:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:10:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:617ms queue-time:0ms Jul 14 22:10:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 22:10:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:10:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:10:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:11:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:11:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:11:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:11:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:11:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:11:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:606ms queue-time:0ms Jul 14 22:11:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 22:11:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:11:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:11:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:12:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:12:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:12:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:12:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:12:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:12:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 22:12:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:12:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:12:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:12:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:692ms queue-time:0ms Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.13.44 2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.14.0 1bd192509acea0b190a7fd1f9680c252 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @epoch=14, @num_updates=0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/configuration/resources/clone[@id='drbd-clone']/primitive[@id='drbd']: Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.14.0) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition 0 aborted by meta_attributes.drbd-meta_attributes 'create': Configuration change | cib=0.14.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/clone[@id='drbd-clone']/primitive[@id='drbd'] complete=true Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-fenced [32268] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create primitive[@id='drbd'] Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-fenced [32268] (cib_devices_update) info: Updating devices to version 0.14.0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is online Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is online Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:0 cannot run anywhere Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:1 cannot run anywhere Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave virsh_node1 (Started el8-a01n01) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave virsh_node2 (Started el8-a01n02) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-15.bz2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_invoke) info: Processing graph 1 (ref=pe_calc-dc-1594764783-31) derived from /var/lib/pacemaker/pengine/pe-input-15.bz2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n01 | action 19 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=19:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=drbd_stop_0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=27, reply=1, notify=0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:drbd action:stop call_id:27 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit drbd named 'drbd' Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 20 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n01: Cancelled | call=25 key=drbd_monitor_60000 confirmed=true Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec_with_unit) debug: Calling StopUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.14.0 2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.14.1 (null) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=1 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=19:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;19:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764783, @last-run=1594764783, @exec-time=0, @queue-time=0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/52, version=0.14.1) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.14.1 2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.14.2 (null) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=2 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=20:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;20:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764783, @last-run=1594764783, @exec-time=0 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.14.2) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-19.raw Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Wrote version 0.14.0 of the CIB to disk (digest: f961ab7ee329b0f7a9c5229a9f0d6bfb) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Isb2IW (digest: /var/lib/pacemaker/cib/cib.U26y3v) Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1941 Jul 14 22:13:03 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=239ms, remaining=99761ms, start_delay=2000ms) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.14.2 2 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.14.3 (null) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=3 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;20:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=27, @rc-code=0, @op-status=0, @exec-time=2199 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 1 action 20 (drbd_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=27 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.14.3) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:27 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop is now complete (elapsed=2246ms, remaining=97754ms): not running (7) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:2246ms queue-time:0ms Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/53) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of stop operation for drbd on el8-a01n01: 0 (ok) | call=27 key=drbd_stop_0 confirmed=true cib-update=53 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.14.3 2 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.14.4 (null) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=4 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;19:1:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=27, @rc-code=0, @op-status=0, @exec-time=2246 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/53, version=0.14.4) Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 1 action 19 (drbd_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=27 Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (run_graph) notice: Transition 1 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-15.bz2): Complete Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 14 22:13:05 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 22:13:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_ping) info: Reporting our current digest to el8-a01n01: d6f800ec4340ab45e7c21c068a77d538 for 0.14.4 (0x55db3e38ea40 0) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.14.4 2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.0 (null) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: -- /cib/configuration/resources/clone[@id='drbd-clone']/primitive[@id='drbd']/meta_attributes[@id='drbd-meta_attributes']/nvpair[@id='drbd-meta_attributes-target-role'] Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @epoch=15, @num_updates=0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.15.0) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition 1 aborted by deletion of nvpair[@id='drbd-meta_attributes-target-role']: Configuration change | cib=0.15.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/clone[@id='drbd-clone']/primitive[@id='drbd']/meta_attributes[@id='drbd-meta_attributes']/nvpair[@id='drbd-meta_attributes-target-role'] complete=true Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-fenced [32268] (update_cib_stonith_devices_v2) info: Updating device list from the cib: (meta) attribute deleted from resource Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-fenced [32268] (cib_devices_update) info: Updating devices to version 0.15.0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is online Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is online Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n02 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave virsh_node1 (Started el8-a01n01) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave virsh_node2 (Started el8-a01n02) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start drbd:0 ( el8-a01n01 ) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Start drbd:1 ( el8-a01n02 ) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-16.bz2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-20.raw Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_invoke) info: Processing graph 2 (ref=pe_calc-dc-1594764793-34) derived from /var/lib/pacemaker/pengine/pe-input-16.bz2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation drbd_start_0 locally on el8-a01n01 | action 17 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=17:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=drbd_start_0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/55) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Wrote version 0.15.0 of the CIB to disk (digest: 5bb60ad37a5f7727a233a50a2a1ce24b) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.0 2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.1 (null) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=1 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=17:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;17:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764793, @last-run=1594764793, @exec-time=0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/55, version=0.15.1) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.vDPMVP (digest: /var/lib/pacemaker/cib/cib.NYCpSU) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=28, reply=1, notify=0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:drbd action:start call_id:28 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit drbd named 'drbd' Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation drbd:1_start_0 on el8-a01n02 | action 19 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec_with_unit) debug: Calling StartUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.1 2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.2 (null) Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=2 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=19:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;19:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764793, @last-run=1594764793, @exec-time=0 Jul 14 22:13:13 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.15.2) Jul 14 22:13:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/1942 Jul 14 22:13:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd start may still be in progress: re-scheduling (elapsed=239ms, remaining=99761ms, start_delay=2000ms) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.2 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.3 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=3 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;19:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=28, @rc-code=0, @op-status=0, @exec-time=2215 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 2 action 19 (drbd_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=28 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_60000 on el8-a01n02 | action 20 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.15.3) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.3 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.4 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=4 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-key=20:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;20:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764796, @exec-time=0 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.15.4) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.4 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.5 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=5 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;20:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=29, @rc-code=0, @op-status=0, @exec-time=3 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.15.5) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 2 action 20 (drbd_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=29 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:28 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd start is now complete (elapsed=2248ms, remaining=97752ms): ok (0) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:28 exit-code:0 exec-time:2248ms queue-time:0ms Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of start operation for drbd on el8-a01n01: 0 (ok) | call=28 key=drbd_start_0 confirmed=true cib-update=56 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/56) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.5 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.6 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=6 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;17:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=28, @rc-code=0, @op-status=0, @exec-time=2248 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/56, version=0.15.6) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 2 action 17 (drbd_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=28 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating monitor operation drbd_monitor_60000 locally on el8-a01n01 | action 18 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=18:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=drbd_monitor_60000 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=29, reply=1, notify=0 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:29 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.6 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.7 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=7 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-key=18:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;18:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764796, @exec-time=0 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.15.7) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:29 exit-code:0 exec-time:7ms queue-time:0ms Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of monitor operation for drbd on el8-a01n01: 0 (ok) | call=29 key=drbd_monitor_60000 confirmed=false cib-update=58 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.7 2 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.8 (null) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=8 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;18:2:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=29, @rc-code=0, @op-status=0, @exec-time=7 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.15.8) Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 2 action 18 (drbd_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=29 Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (run_graph) notice: Transition 2 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-16.bz2): Complete Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 14 22:13:16 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 22:13:21 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 5847f7496bbf44bdca749ca35de63ec1 for 0.15.8 (0x55db3dd5f7a0 0) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was member) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (handle_shutdown_request) info: Creating shutdown request for el8-a01n02 (state=S_IDLE) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594764807 | from el8-a01n01 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.8 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.9 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=9 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/4, version=0.15.9) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition 2 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.15.9 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=true Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is online Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is shutting down Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource hypervisor:1 cannot run anywhere Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:1 cannot run anywhere Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n02 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogNodeActions) notice: * Shutdown el8-a01n02 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave virsh_node1 (Started el8-a01n01) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Move virsh_node2 ( el8-a01n02 -> el8-a01n01 ) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop hypervisor:1 ( el8-a01n02 ) due to node availability Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave drbd:0 (Started el8-a01n01) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-17.bz2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_invoke) info: Processing graph 3 (ref=pe_calc-dc-1594764807-39) derived from /var/lib/pacemaker/pengine/pe-input-17.bz2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n02 | action 9 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 on el8-a01n02 | action 14 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 21 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.9 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.10 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=10 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=9:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;9:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764807, @last-run=1594764807, @exec-time=0 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/35, version=0.15.10) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.10 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.11 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=11 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=14:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;14:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764807, @last-run=1594764807, @exec-time=0 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/36, version=0.15.11) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.11 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.12 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=12 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;9:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=31, @rc-code=0, @op-status=0, @exec-time=1 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/37, version=0.15.12) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 3 action 9 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=31 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n01 | action 10 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=10:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node2_start_0 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=30, reply=1, notify=0 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:virsh_node2 action:start call_id:30 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.12 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.13 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=13 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=21:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;21:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764807, @last-run=1594764807, @exec-time=0 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/38, version=0.15.13) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/60) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.13 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.14 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=14 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=10:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;10:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764807, @last-run=1594764807, @queue-time Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/60, version=0.15.14) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemakerd [32266] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_shutdown_worker) notice: Shutting down Pacemaker Jul 14 22:13:27 el8-a01n01.digimer.ca pacemakerd [32266] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 32273 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_log) warning: Input I_SHUTDOWN received in state S_TRANSITION_ENGINE from crm_shutdown Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n01) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_POLICY_ENGINE) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594764807 | from el8-a01n01 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.14 2 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.15 (null) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=15 Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.15.15) Jul 14 22:13:27 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) notice: Transition 3 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.15.15 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=false Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) info: finished - rsc:virsh_node2 action:start call_id:30 exit-code:0 exec-time:652ms queue-time:0ms Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of start operation for virsh_node2 on el8-a01n01: 0 (ok) | call=30 key=virsh_node2_start_0 confirmed=true cib-update=61 Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61) Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.15 2 Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.16 (null) Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=16 Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;10:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=30, @rc-code=0, @op-status=0, @exec-time=652 Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 3 action 10 (virsh_node2_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=30 Jul 14 22:13:28 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.15.16) Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.16 2 Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.17 (null) Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=17 Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;14:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=33, @rc-code=0, @op-status=0, @exec-time=2219 Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/39, version=0.15.17) Jul 14 22:13:29 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 3 action 14 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=33 Jul 14 22:13:34 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_ping) info: Reporting our current digest to el8-a01n01: a76984cee9df38db885f8565e9ac76c7 for 0.15.17 (0x55db3dd5f7a0 0) Jul 14 22:13:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:13:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:13:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:13:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:13:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:13:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:664ms queue-time:0ms Jul 14 22:14:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 22:14:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:14:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:14:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:29 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:14:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 22:14:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:14:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:14:41 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:14:43 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 22:14:44 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:639ms queue-time:0ms Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.17 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.18 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=18 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=2:198;21:3:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=35, @rc-code=198, @op-status=2, @exec-time=99990 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.15.19) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.19 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.20 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=20 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.15.20) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition 3 aborted by status-2-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change | cib=0.15.19 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (abort_transition_graph) info: Transition 3 aborted by status-2-last-failure-drbd.stop_0 doing create last-failure-drbd#stop_0=1594764907: Transient attribute change | cib=0.15.20 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is shutting down Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is shutting down Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 22:13:27 2020 | rc=198 id=drbd_last_failure_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 22:13:27 2020 | rc=198 id=drbd_last_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Stopped: [ el8-a01n02 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource virsh_node1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource virsh_node2 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource hypervisor:0 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource hypervisor:1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:0 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (stage6) warning: Scheduling Node el8-a01n02 for STONITH Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogNodeActions) notice: * Shutdown el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there' Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop virsh_node1 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop virsh_node2 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:1 (Stopped) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) warning: Calculated transition 4 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-4.bz2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (handle_response) info: pe_calc calculation pe_calc-dc-1594764907-45 is obsolete Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n01 is shutting down Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (determine_online_status) info: Node el8-a01n02 is shutting down Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 22:13:27 2020 | rc=198 id=drbd_last_failure_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 22:13:27 2020 | rc=198 id=drbd_last_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 1 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (unpack_node_loop) info: Node 2 is already processed Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Stopped: [ el8-a01n02 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (short_print) info: Started: [ el8-a01n01 ] Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource virsh_node1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource virsh_node2 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource hypervisor:0 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource hypervisor:1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:0 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_color) info: Resource drbd:1 cannot run anywhere Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (stage6) warning: Scheduling Node el8-a01n02 for STONITH Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogNodeActions) notice: * Shutdown el8-a01n01 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there' Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop virsh_node1 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop virsh_node2 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogActions) info: Leave hypervisor:1 (Stopped) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-schedulerd[32271] (pcmk__log_transition_summary) warning: Calculated transition 5 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-5.bz2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_te_invoke) info: Processing graph 5 (ref=pe_calc-dc-1594764907-46) derived from /var/lib/pacemaker/pengine/pe-warn-5.bz2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 locally on el8-a01n01 | action 6 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=6:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node1_stop_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=32, reply=1, notify=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:virsh_node1 action:stop call_id:32 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/65) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) info: finished - rsc:virsh_node1 action:stop call_id:32 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n01 | action 7 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=7:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=virsh_node2_stop_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=33, reply=1, notify=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:33 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/66) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:33 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=5 timeout=60000 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.20 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.21 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=21 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=6:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;6:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764907, @last-run=1594764907, @exec-time=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (handle_request) notice: Client pacemaker-controld.32273.16321daa wants to fence (reboot) 'el8-a01n02' with device '(any)' Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=f775778c-fc66-48cb-b74e-926c5586336e state=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/65, version=0.15.21) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.21 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.22 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=22 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=7:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;7:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764907, @last-run=1594764907, @exec-time=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) f775778c-fc66-48cb-b74e-926c5586336e Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) info: Result of monitor operation for virsh_node1 on el8-a01n01: Cancelled | call=22 key=virsh_node1_monitor_60000 confirmed=true Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/66, version=0.15.22) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) f775778c-fc66-48cb-b74e-926c5586336e Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.32273|id=f775778c-fc66-48cb-b74e-926c5586336e Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.32273 (72s, 0s) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok) | call=32 key=virsh_node1_stop_0 confirmed=true cib-update=67 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/67) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-fenced [32268] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02' Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.22 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.23 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=23 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;6:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=32, @rc-code=0, @op-status=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/67, version=0.15.23) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/68) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.23 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.24 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=24 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;7:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=33, @rc-code=0, @op-status=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/68, version=0.15.24) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n01: 0 (ok) | call=33 key=virsh_node2_stop_0 confirmed=true cib-update=68 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 5 action 6 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=32 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 5 action 7 (virsh_node2_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=33 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n01 | action 8 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=8:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=hypervisor_stop_0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=35, reply=1, notify=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/69) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:hypervisor action:stop call_id:35 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit libvirtd named 'hypervisor' Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.24 2 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.25 (null) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=25 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=8:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;8:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764907, @last-run=1594764907, @exec-time=0 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/69, version=0.15.25) Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec_with_unit) debug: Calling StopUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n01: Cancelled | call=24 key=hypervisor_monitor_60000 confirmed=true Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/2046 Jul 14 22:15:07 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: hypervisor stop may still be in progress: re-scheduling (elapsed=235ms, remaining=99765ms, start_delay=2000ms) Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:35 Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: hypervisor stop is now complete (elapsed=2244ms, remaining=97756ms): not running (7) Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:35 exit-code:0 exec-time:2244ms queue-time:0ms Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok) | call=35 key=hypervisor_stop_0 confirmed=true cib-update=70 Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/70) Jul 14 22:15:09 el8-a01n01.digimer.ca pacemaker-fenced [32268] (log_operation) notice: Operation 'reboot' [60289] (call 2 from pacemaker-controld.32273) for host 'el8-a01n02' with device 'virsh_node2' returned: 0 (OK) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_quorum_notification) info: Quorum retained | membership=179 members=1 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Cluster node el8-a01n02 is now lost (was member) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 2: el8-a01n02 (node 2 pid 1815) left via cluster exit Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n02 (node 2 pid 1818) left via cluster exit Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (pcmk_cpg_membership) info: Group cib event 2: el8-a01n01 (node 1 pid 32267) is member Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_remove_voter) notice: Lost attribute writer el8-a01n02 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_start_election_if_needed) info: Starting an election to determine the writer Jul 14 22:15:10 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_quorum_notification) info: Quorum retained | membership=179 members=1 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 14 22:15:10 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 2: el8-a01n02 (node 2 pid 1814) left via cluster exit Jul 14 22:15:10 el8-a01n01.digimer.ca pacemakerd [32266] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_peer_remove) notice: Removing all el8-a01n02 attributes for peer loss Jul 14 22:15:10 el8-a01n01.digimer.ca pacemakerd [32266] (pcmk_cpg_membership) info: Group pacemakerd event 2: el8-a01n01 (node 1 pid 32266) is member Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n01 (node 1 pid 32270) is member Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (election_check) info: election-attrd won by local node Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (write_attribute) info: Sent CIB request 4 with 1 change for shutdown (id n/a, set n/a) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.25 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.26 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=26 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;8:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=35, @rc-code=0, @op-status=0, @exec-time=2244 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/70, version=0.15.26) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/4) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 2: el8-a01n02 (node 2 pid 1816) left via cluster exit Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_graph_event) info: Transition 5 action 8 (hypervisor_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=35 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n02 (node 2 pid 1820) left via cluster exit Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now offline Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (peer_update_callback) info: Node el8-a01n02 is no longer a peer | DC=true old=0x4000000 new=0x0000000 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (controld_delete_node_state) info: Deleting transient attributes for node el8-a01n02 (via CIB call 76) | xpath=//node_state[@uname='el8-a01n02']/transient_attributes Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n01 (node 1 pid 32273) is member Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/71) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (crm_reap_dead_member) info: Removing node with name el8-a01n02 and id 2 from membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=el8-a01n02 from the membership cache Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (pcmk_cpg_membership) info: Group stonith-ng event 2: el8-a01n01 (node 1 pid 32268) is member Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/4, version=0.15.26) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_cib_callback) info: CIB update 4 result for shutdown: OK | rc=0 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-attrd [32270] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594764807 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-fenced [32268] (remote_op_done) notice: Operation 'reboot' targeting el8-a01n02 on el8-a01n01 for pacemaker-controld.32273@el8-a01n01.f775778c: OK Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (tengine_stonith_callback) notice: Stonith operation 2/5:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af: OK (0) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (tengine_stonith_callback) info: Stonith operation 2 for el8-a01n02 passed Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (controld_delete_node_state) info: Deleting all state for node el8-a01n02 (via CIB call 79) | xpath=//node_state[@uname='el8-a01n02']/* Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (tengine_stonith_notify) notice: Peer el8-a01n02 was terminated (reboot) by el8-a01n01 on behalf of pacemaker-controld.32273: OK | initiator=el8-a01n01 ref=f775778c-fc66-48cb-b74e-926c5586336e Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (controld_delete_node_state) info: Deleting all state for node el8-a01n02 (via CIB call 81) | xpath=//node_state[@uname='el8-a01n02']/* Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n01 | action 13 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_finished) debug: finished - rsc:drbd action:monitor call_id:29 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=0, reply=1, notify=0 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.26 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.27 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=27 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=peer_update_callback Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/71, version=0.15.27) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/74) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (do_lrm_rsc_op) info: Performing key=13:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af op=drbd_stop_0 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from ee3d8d6e-17ff-4c5b-a03a-b568ee2dd1e1: rc=37, reply=1, notify=0 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) info: executing - rsc:drbd action:stop call_id:37 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit drbd named 'drbd' Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec_with_unit) debug: Calling StopUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n01: Cancelled | call=29 key=drbd_monitor_60000 confirmed=true Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/75) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes to all (origin=local/crmd/76) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/77) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/74, version=0.15.27) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.27 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.28 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=28 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update, @expected=down Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @in_ccm=false, @crm-debug-origin=post_cache_update, @expected=down Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/75, version=0.15.28) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.28 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.29 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2'] Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=29 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n01/crmd/76, version=0.15.29) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.29 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.30 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=30 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crmd=offline, @crm-debug-origin=peer_update_callback Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/77, version=0.15.30) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/78) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/* to all (origin=local/crmd/79) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/80) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/* to all (origin=local/crmd/81) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/82) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.30 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.31 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=31 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=send_stonith_update, @join=down Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/78, version=0.15.31) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.31 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.32 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=32 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/*: OK (rc=0, origin=el8-a01n01/crmd/79, version=0.15.32) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/80, version=0.15.32) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/*: OK (rc=0, origin=el8-a01n01/crmd/81, version=0.15.32) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: --- 0.15.32 2 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: Diff: +++ 0.15.33 (null) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib: @num_updates=33 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=13:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @transition-magic=-1:193;13:5:0:ae37b40f-84ce-4cb6-b3a6-7dd8a73a65af, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594764910, @last-run=1594764910, @exec-time=0 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/82, version=0.15.33) Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (cib_fencing_updated) info: Fencing update 78 for el8-a01n02: complete Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-controld [32273] (cib_fencing_updated) info: Fencing update 80 for el8-a01n02: complete Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/2047 Jul 14 22:15:10 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=197ms, remaining=99803ms, start_delay=2000ms) Jul 14 22:15:12 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:12 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:12 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:12 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=2202ms, remaining=97798ms, start_delay=2000ms) Jul 14 22:15:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:14 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=4243ms, remaining=95757ms, start_delay=2000ms) Jul 14 22:15:15 el8-a01n01.digimer.ca pacemaker-based [32267] (cib_process_ping) info: Reporting our current digest to el8-a01n01: a943c9f319091238e9c520a05c99839d for 0.15.33 (0x55db3e3826c0 0) Jul 14 22:15:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:16 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=6248ms, remaining=93752ms, start_delay=2000ms) Jul 14 22:15:18 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:18 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:18 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:18 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=8253ms, remaining=91747ms, start_delay=2000ms) Jul 14 22:15:20 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:20 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:20 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:20 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=10257ms, remaining=89743ms, start_delay=2000ms) Jul 14 22:15:22 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:22 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:22 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:22 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=12264ms, remaining=87736ms, start_delay=2000ms) Jul 14 22:15:24 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:24 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:24 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:24 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=14270ms, remaining=85730ms, start_delay=2000ms) Jul 14 22:15:26 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:26 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:26 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:26 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=16277ms, remaining=83723ms, start_delay=2000ms) Jul 14 22:15:28 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:28 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:28 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:28 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=18280ms, remaining=81720ms, start_delay=2000ms) Jul 14 22:15:30 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:30 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:30 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:30 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=20293ms, remaining=79707ms, start_delay=2000ms) Jul 14 22:15:32 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:32 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:32 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:32 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=22303ms, remaining=77697ms, start_delay=2000ms) Jul 14 22:15:34 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:34 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:34 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:34 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=24317ms, remaining=75683ms, start_delay=2000ms) Jul 14 22:15:36 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:36 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:36 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:36 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=26328ms, remaining=73672ms, start_delay=2000ms) Jul 14 22:15:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:38 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=28357ms, remaining=71643ms, start_delay=2000ms) Jul 14 22:15:40 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:40 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:40 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:40 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=30367ms, remaining=69633ms, start_delay=2000ms) Jul 14 22:15:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:42 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=32372ms, remaining=67628ms, start_delay=2000ms) Jul 14 22:15:44 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:44 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:44 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:44 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=34375ms, remaining=65625ms, start_delay=2000ms) Jul 14 22:15:46 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:46 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:46 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:46 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=36383ms, remaining=63617ms, start_delay=2000ms) Jul 14 22:15:48 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:48 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:48 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:48 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=38386ms, remaining=61614ms, start_delay=2000ms) Jul 14 22:15:50 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:50 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:50 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:50 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=40393ms, remaining=59607ms, start_delay=2000ms) Jul 14 22:15:52 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:52 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:52 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:52 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=42398ms, remaining=57602ms, start_delay=2000ms) Jul 14 22:15:54 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:54 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:54 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:54 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=44408ms, remaining=55592ms, start_delay=2000ms) Jul 14 22:15:56 el8-a01n01.digimer.ca pacemaker-execd [32269] (log_execute) debug: executing - rsc:drbd action:monitor call_id:37 Jul 14 22:15:56 el8-a01n01.digimer.ca pacemaker-execd [32269] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 22:15:56 el8-a01n01.digimer.ca pacemaker-execd [32269] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 22:15:56 el8-a01n01.digimer.ca pacemaker-execd [32269] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=46411ms, remaining=53589ms, start_delay=2000ms)