[root@el8-a01n01 ~]# cat /var/log/pacemaker/el8-a01n01/pacemaker.log.drbd-restart-to-crash Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (get_cluster_type) info: Detected an active 'corosync' cluster Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (mcp_read_config) info: Reading configure for stack: corosync Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (main) notice: Starting Pacemaker 2.0.3-5.el8_2.1 | build=4b1f869f0f features: generated-manpages agent-manpages ncurses libqb-logging libqb-ipc systemd nagios corosync-native atomic-attrd acls Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (main) info: Maximum core file size is: 18446744073709551615 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (qb_ipcs_us_publish) info: server name: pacemakerd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to lrmd IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to cib_ro IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to crmd IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to pengine IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk__ipc_is_authentic_process_active) info: Could not connect to stonith-ng IPC: Connection refused Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Created entry 81dfc4f1-02f0-435d-8154-a23bf123823b/0x558d0df928c0 for node el8-a01n01/1 (1 total) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (cluster_connect_quorum) warning: Quorum lost Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Created entry c378e0a7-0e78-4209-a7c0-499064bd0716/0x558d0df94860 for node el8-a01n02/2 (2 total) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Using uid=189 and group=189 for process pacemaker-based Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6051 for process pacemaker-based Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6052 for process pacemaker-fenced Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6053 for process pacemaker-execd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Using uid=189 and group=189 for process pacemaker-attrd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6054 for process pacemaker-attrd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Using uid=189 and group=189 for process pacemaker-schedulerd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6055 for process pacemaker-schedulerd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Using uid=189 and group=189 for process pacemaker-controld Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (start_child) info: Forked child 6056 for process pacemaker-controld Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (main) notice: Pacemaker daemon successfully started and accepting connections Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_quorum_notification) info: Quorum still lost | membership=162 members=1 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (crm_reap_unseen_nodes) info: State of node el8-a01n02[2] is still unknown Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_cpg_membership) info: Group pacemakerd event 0: node 1 pid 6050 joined via cpg_join Jul 14 21:52:20 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_cpg_membership) info: Group pacemakerd event 0: el8-a01n01 (node 1 pid 6050) is member Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-execd [6053] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-execd [6053] (main) notice: Starting Pacemaker local executor Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_ipcs_us_publish) info: server name: lrmd Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-execd [6053] (main) notice: Pacemaker local executor successfully started and accepting connections Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (main) notice: Starting Pacemaker scheduler Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (qb_ipcs_us_publish) info: server name: pengine Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (main) notice: Pacemaker scheduler successfully started and accepting connections Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (main) notice: Starting Pacemaker fencer Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (main) notice: Starting Pacemaker CIB manager Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Created entry b880f033-a364-4fd5-b74b-07b7fa1c03c1/0x5653eed71d60 for node el8-a01n01/1 (1 total) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-attrd [6054] (main) notice: Starting Pacemaker node attribute manager Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-fenced [6052] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (retrieveCib) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-controld [6056] (main) notice: Starting Pacemaker controller Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) info: Input I_STARTUP received in state S_STARTING from crmd_init Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Created entry 16e38ebd-b2a7-4fbd-b06e-4d33ecf0c8d6/0x563bd6586c00 for node el8-a01n01/1 (1 total) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (qb_ipcs_us_publish) info: server name: cib_ro Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (qb_ipcs_us_publish) info: server name: cib_rw Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (qb_ipcs_us_publish) info: server name: cib_shm Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (main) notice: Pacemaker CIB manager successfully started and accepting connections Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (pcmk_cpg_membership) info: Group cib event 0: node 1 pid 6051 joined via cpg_join Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (pcmk_cpg_membership) info: Group cib event 0: el8-a01n01 (node 1 pid 6051) is member Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-15.raw Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_write_with_digest) info: Wrote version 0.13.0 of the CIB to disk (digest: 104d39c1b15a7a8d4fc8d317785f3f93) Jul 14 21:52:20 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.vVRNsY (digest: /var/lib/pacemaker/cib/cib.1pOqgd) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (main) info: CIB connection active Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Created entry ab98b779-8aea-4113-b448-95d76468bf29/0x55c354e20a20 for node el8-a01n01/1 (1 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (main) info: Cluster connection active Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_erase_attrs) info: Clearing transient attributes from CIB | xpath=//node_state[@uname='el8-a01n01']/transient_attributes Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (qb_ipcs_us_publish) info: server name: attrd Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (main) notice: Pacemaker node attribute manager successfully started and accepting connections Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes to all (origin=local/attrd/2) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (setup_cib) info: Watching for stonith topology changes Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (qb_ipcs_us_publish) info: server name: stonith-ng Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (main) notice: Pacemaker fencer successfully started and accepting connections Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (pcmk_cpg_membership) info: Group stonith-ng event 0: node 1 pid 6052 joined via cpg_join Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (pcmk_cpg_membership) info: Group stonith-ng event 0: el8-a01n01 (node 1 pid 6052) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (init_cib_cache_cb) info: Updating device list from the cib: init Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (cib_devices_update) info: Updating devices to version 0.13.0 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (get_cluster_type) info: Verifying cluster type: 'corosync' Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (get_cluster_type) info: Assuming an active 'corosync' cluster Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Created entry e98b1b68-2110-443a-b851-43406298de36/0x55f25c909e20 for node el8-a01n01/1 (1 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Node 1 is now known as el8-a01n01 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Cluster node el8-a01n01 is now in unknown state Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Node 1 has uuid 1 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Node el8-a01n01 is now a peer | DC= old=0x0000000 new=0x4000000 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (init_cs_connection_once) info: Connection to 'corosync': established Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (cluster_connect_quorum) warning: Quorum lost Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Created entry 8bd55a82-ce1c-4ca8-8981-5bc419c15ae1/0x55f25c90c700 for node el8-a01n02/2 (2 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Cluster node el8-a01n02 is now in unknown state Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_ha_control) info: Connected to the cluster Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (lrmd_ipc_connect) info: Connecting to executor Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/4) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (crm_client_new) debug: Connecting 0x565394bc4fb0 for uid=189 gid=189 pid=6056 id=29872d72-cf4e-4597-93c4-86f8b71374d4 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (handle_new_connection) debug: IPC credentials authenticated (/dev/shm/qb-6053-6056-9-WwUHL3/qb) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_ipcs_shm_connect) debug: connecting to client [6056] Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed register operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_control) info: Connection to the executor established Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_started) info: Delaying start, no membership data (0000000000100000) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_started) info: Delaying start, no membership data (0000000000100000) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (pcmk_quorum_notification) info: Quorum still lost | membership=162 members=1 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Cluster node el8-a01n01 is now member (was in unknown state) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_reap_unseen_nodes) info: State of node el8-a01n02[2] is still unknown Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_started) info: Delaying start, Config not read (0000000000000040) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (qb_ipcs_us_publish) info: server name: crmd Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_started) notice: Pacemaker controller successfully started and accepting connections Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) info: Input I_PENDING received in state S_STARTING from do_started Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_cpg_membership) info: Group pacemakerd event 1: node 2 pid 1596 joined via cluster join Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n01 (node 1 pid 6050) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n02 (node 2 pid 1596) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining, last=7): Try again (6) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_quorum_notification) notice: Quorum acquired | membership=166 members=2 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemakerd [6050] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (pcmk_cpg_membership) info: Group attrd event 0: node 1 pid 6054 joined via cpg_join Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (pcmk_cpg_membership) info: Group cib event 1: node 2 pid 1597 joined via cluster join Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n01 (node 1 pid 6051) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n01 (node 1 pid 6054) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Created entry ee035951-ee4d-40a4-a722-400d6905e854/0x55c354e27060 for node el8-a01n02/2 (2 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n02 (node 2 pid 1600) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Created entry 7c226def-c52b-4818-8021-a87a2356d821/0x563bd6613490 for node el8-a01n02/2 (2 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n02 (node 2 pid 1597) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes: OK (rc=0, origin=el8-a01n01/attrd/2, version=0.13.0) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/4, version=0.13.0) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n02/attrd/2, version=0.13.0) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2 | from el8-a01n01 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_start_election_if_needed) info: Starting an election to determine the writer Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2 | from el8-a01n02 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (election_count_vote) info: election-attrd round 1 (owner node ID 2) pass: vote from el8-a01n02 (Uptime) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (election_check) info: election-attrd won by local node Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-attrd [6054] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/4, version=0.13.0) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 6096 exited with rc=0 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (stonith_device_register) notice: Added 'virsh_node1' to the device list (1 active devices) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (stonith_device_register) notice: Added 'virsh_node2' to the device list (2 active devices) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 2 pid 1598 joined via cluster join Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n01 (node 1 pid 6052) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Created entry b9377532-f6fc-4e21-a8ea-32622712e98c/0x5653eee79000 for node el8-a01n02/2 (2 total) Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Node 2 is now known as el8-a01n02 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_get_peer) info: Node 2 has uuid 2 Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n02 (node 2 pid 1598) is member Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 21:52:21 el8-a01n01.digimer.ca pacemaker-fenced [6052] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (pcmk_quorum_notification) notice: Quorum acquired | membership=166 members=2 Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Cluster node el8-a01n02 is now member (was in unknown state) Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (pcmk_cpg_membership) info: Group crmd event 0: node 1 pid 6056 joined via cpg_join Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n01 (node 1 pid 6056) is member Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n02 (node 2 pid 1602) is member Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (peer_update_callback) info: Node el8-a01n02 is now a peer | DC= old=0x0000000 new=0x4000000 Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 30 seconds Jul 14 21:52:22 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_connect_stonith) notice: Fencer successfully connected Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_timer_popped) info: Election Trigger just popped | input=I_DC_TIMEOUT time=20000ms Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) warning: Input I_DC_TIMEOUT received in state S_PENDING from crm_timer_popped Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_PENDING -> S_ELECTION | input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (election_check) info: election-DC won by local node Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_te_control) info: Registering TE UUID: 7fee6bf5-1c09-43c8-93d7-f6f42af0e11c Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (set_graph_functions) info: Setting custom graph functions Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_dc_takeover) info: Taking over DC status for this partition Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_readwrite) info: We are now in R/W mode Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/8) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/10) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/12) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/8, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/10, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/12, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (join_make_offer) info: Making join-1 offers based on membership event 166 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (join_make_offer) info: Sending join-1 offer to el8-a01n01 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/14) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (join_make_offer) info: Sending join-1 offer to el8-a01n02 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/14, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_dc_join_offer_all) info: Waiting on join-1 requests from 2 outstanding nodes Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (update_dc) info: Set DC to el8-a01n01 (3.3.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_expected) info: update_dc: Node el8-a01n01[1] - expected state is now member (was (null)) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_expected) info: do_dc_join_filter_offer: Node el8-a01n02[2] - expected state is now member (was (null)) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_replace) info: Digest matched on replace from el8-a01n01: e66cd993ae6725062fe95afbc6bd62d5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_replace) info: Replaced 0.13.0 with 0.13.0 from el8-a01n01 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=el8-a01n01/crmd/17, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/18) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (controld_delete_node_state) info: Deleting resource history for node el8-a01n01 (via CIB call 20) | xpath=//node_state[@uname='el8-a01n01']/lrm Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/18, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/19) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (controld_delete_node_state) info: Deleting resource history for node el8-a01n02 (via CIB call 22) | xpath=//node_state[@uname='el8-a01n02']/lrm Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm to all (origin=local/crmd/20) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/21) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/19, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm: OK (rc=0, origin=el8-a01n01/crmd/20, version=0.13.0) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.0 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.1 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/21, version=0.13.1) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (abort_transition_graph) info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm to all (origin=local/crmd/22) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/23) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm: OK (rc=0, origin=el8-a01n01/crmd/22, version=0.13.1) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.1 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.2 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/23, version=0.13.2) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/26) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/28) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/26, version=0.13.2) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.2 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.3 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=3 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition, @join=member, @expected=member Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition, @join=member, @expected=member Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/27, version=0.13.3) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.3 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.4 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=4, @dc-uuid=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/28, version=0.13.4) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n01 is online Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n02 is online Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ] Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ] Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:0 on el8-a01n01 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:1 on el8-a01n02 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n02 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start virsh_node1 ( el8-a01n01 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start virsh_node2 ( el8-a01n02 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start hypervisor:0 ( el8-a01n01 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start hypervisor:1 ( el8-a01n02 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start drbd:0 ( el8-a01n01 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Start drbd:1 ( el8-a01n02 ) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pcmk__log_transition_summary) notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-9.bz2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_te_invoke) info: Processing graph 0 (ref=pe_calc-dc-1594763562-10) derived from /var/lib/pacemaker/pengine/pe-input-9.bz2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n02 | action 5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-16.raw Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n01 | action 1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node1' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node1' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=1:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=virsh_node1_monitor_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_write_with_digest) info: Wrote version 0.13.0 of the CIB to disk (digest: fc0fc716ef8c227358159deeb0d8258c) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.rvY7DK (digest: /var/lib/pacemaker/cib/cib.mKtSV2) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.4 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.5 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/8, version=0.13.5) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.5 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.6 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=6 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;5:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=5, @rc-code=7, @op-status=0, @exec-time=4, @queue-time=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/9, version=0.13.6) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 7821 exited with rc=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/30) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=5, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (stonith_api_signon) debug: Attempting fencer connection by pacemaker-execd with mainloop Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.6 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.7 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=7 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/30, version=0.13.7) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n02 | action 6 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n01 | action 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (stonith_api_signon) debug: Connection to fencer by pacemaker-execd succeeded (registration token: 12bdea4b-c279-4414-9c2e-dcf1c44cde18) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:5 exit-code:7 exec-time:4ms queue-time:0ms Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node2' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.7 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.8 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=8 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/10, version=0.13.8) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node2' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=2:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=virsh_node2_monitor_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=9, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:9 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:9 exit-code:7 exec-time:0ms queue-time:0ms Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.8 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.9 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=9 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;6:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=9, @rc-code=7, @op-status=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/11, version=0.13.9) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.9 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.10 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=10 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/31, version=0.13.10) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 5 (virsh_node1_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 6 (virsh_node2_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=9 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of probe operation for virsh_node1 on el8-a01n01: 7 (not running) | call=5 key=virsh_node1_monitor_0 confirmed=true cib-update=32 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/32) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.10 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.11 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=11 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;1:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=5, @rc-code=7, @op-status=0, @exec-time=4 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/32, version=0.13.11) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/33) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.11 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.12 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=12 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;2:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=9, @rc-code=7, @op-status=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/33, version=0.13.12) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of probe operation for virsh_node2 on el8-a01n01: 7 (not running) | call=9 key=virsh_node2_monitor_0 confirmed=true cib-update=33 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 1 (virsh_node1_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=5 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 2 (virsh_node2_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=9 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 locally on el8-a01n01 | action 9 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=9:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=virsh_node1_start_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/34) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=10, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:virsh_node1 action:start call_id:10 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.12 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.13 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=13 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=9:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;9:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/34, version=0.13.13) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 on el8-a01n02 | action 11 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_0 locally on el8-a01n01 | action 3 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor:0' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_rsc_register) info: Cached agent information for 'hypervisor' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=3:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=hypervisor_monitor_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/35) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=15, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:15 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.13 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.14 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=14 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=11:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;11:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/12, version=0.13.14) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.14 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.15 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=15 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/35, version=0.13.15) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_0 on el8-a01n02 | action 7 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.15 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.16 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=16 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/13, version=0.13.16) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:15 exit-code:7 exec-time:40ms queue-time:0ms Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of probe operation for hypervisor on el8-a01n01: 7 (not running) | call=15 key=hypervisor_monitor_0 confirmed=true cib-update=36 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.16 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.17 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=17 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;3:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=15, @rc-code=7, @op-status=0, @exec-time=40 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/36, version=0.13.17) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 3 (hypervisor_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=15 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_0 locally on el8-a01n01 | action 4 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'drbd' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_get_rsc_info) info: Agent information for 'drbd:0' not in cache Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_rsc_register) info: Cached agent information for 'drbd' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=4:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=drbd_monitor_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/37) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=20, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:20 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.17 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.18 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=18 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/37, version=0.13.18) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:20 exit-code:7 exec-time:10ms queue-time:0ms Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of probe operation for drbd on el8-a01n01: 7 (not running) | call=20 key=drbd_monitor_0 confirmed=true cib-update=38 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/38) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.18 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.19 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=19 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;4:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=20, @rc-code=7, @op-status=0, @exec-time=10 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/38, version=0.13.19) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 4 (drbd_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=20 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.19 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.20 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=20 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;7:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=15, @rc-code=7, @op-status=0, @exec-time=52 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 7 (hypervisor_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=15 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_0 on el8-a01n02 | action 8 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating start operation hypervisor:0_start_0 locally on el8-a01n01 | action 13 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=13:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=hypervisor_start_0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/14, version=0.13.20) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/39) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=21, reply=1, notify=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:hypervisor action:start call_id:21 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit libvirtd named 'hypervisor' Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.20 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.21 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=21 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=13:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;13:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/39, version=0.13.21) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.21 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.22 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=22 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/15, version=0.13.22) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.22 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.23 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=23 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;8:0:7:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=20, @rc-code=7, @op-status=0, @exec-time=12 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/16, version=0.13.23) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 8 (drbd_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=20 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating start operation hypervisor:1_start_0 on el8-a01n02 | action 15 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec_with_unit) debug: Calling StartUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.23 2 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.24 (null) Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=24 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=15:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;15:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 14 21:52:42 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/17, version=0.13.24) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/982 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: hypervisor start may still be in progress: re-scheduling (elapsed=222ms, remaining=99778ms, start_delay=2000ms) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) info: finished - rsc:virsh_node1 action:start call_id:10 exit-code:0 exec-time:784ms queue-time:0ms Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of start operation for virsh_node1 on el8-a01n01: 0 (ok) | call=10 key=virsh_node1_start_0 confirmed=true cib-update=40 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/40) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.24 2 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.25 (null) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=25 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;9:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=10, @rc-code=0, @op-status=0, @exec-time=784 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/40, version=0.13.25) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 9 (virsh_node1_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=10 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_60000 locally on el8-a01n01 | action 10 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=10:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=virsh_node1_monitor_60000 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.25 2 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.26 (null) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=26 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;11:0:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=10, @rc-code=0, @op-status=0, @exec-time=782 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=22, reply=1, notify=0 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/18, version=0.13.26) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/41) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 0 action 11 (virsh_node2_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=10 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_60000 on el8-a01n02 | action 12 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.26 2 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.27 (null) Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=27 Jul 14 21:52:43 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 21:52:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_procfs_pid_of) info: Found pacemaker-based active as process 6051 Jul 14 21:52:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (throttle_send_command) info: New throttle mode: negligible load (was undetermined) Jul 14 21:52:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_ping) info: Reporting our current digest to el8-a01n01: ede44ae72beac282a4507ae397160828 for 0.13.44 (0x563bd67251f0 0) Jul 14 21:53:44 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 21:53:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 21:53:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 21:53:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:53:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:53:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:632ms queue-time:0ms Jul 14 21:53:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 21:53:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:53:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:53:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 21:54:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:643ms queue-time:0ms Jul 14 21:54:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 21:54:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:54:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:54:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:55:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000 Jul 14 21:55:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 21:55:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:55:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:55:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:22 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:652ms queue-time:0ms Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:7 exec-time:0ms queue-time:0ms Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of monitor operation for drbd on el8-a01n01: 7 (not running) | call=25 key=drbd_monitor_60000 confirmed=false cib-update=50 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/50) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.44 2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.45 (null) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=45 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']: S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_client_update) info: Expanded fail-count-drbd#monitor_60000=value++ to 1 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting fail-count-drbd#monitor_60000[el8-a01n01]: (unset) -> 1 | from el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (write_attribute) info: Sent CIB request 4 with 1 change for fail-count-drbd#monitor_60000 (id n/a, set n/a) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/4) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting last-failure-drbd#monitor_60000[el8-a01n01]: (unset) -> 1594763746 | from el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (write_attribute) info: Sent CIB request 5 with 1 change for last-failure-drbd#monitor_60000 (id n/a, set n/a) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.45 2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.46 (null) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=46 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/4, version=0.13.46) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: CIB update 4 result for fail-count-drbd#monitor_60000: OK | rc=0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: * fail-count-drbd#monitor_60000[el8-a01n01]=1 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/5) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.46 2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.47 (null) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=47 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/5, version=0.13.47) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: CIB update 5 result for last-failure-drbd#monitor_60000: OK | rc=0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: * last-failure-drbd#monitor_60000[el8-a01n01]=1594763746 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (abort_transition_graph) info: Transition 0 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.13.46 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=true Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (abort_transition_graph) info: Transition 0 aborted by status-1-last-failure-drbd.monitor_60000 doing create last-failure-drbd#monitor_60000=1594763746: Transient attribute change | cib=0.13.47 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n01 is online Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n02 is online Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_rsc_op_failure) warning: Unexpected result (not running) was recorded for monitor of drbd:0 on el8-a01n01 at Jul 14 21:55:46 2020 | rc=7 id=drbd_last_failure_0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n02 ] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node1 (Started el8-a01n01) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node2 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Recover drbd:0 ( el8-a01n01 ) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave drbd:1 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-10.bz2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (handle_response) info: pe_calc calculation pe_calc-dc-1594763746-32 is obsolete Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n01 is active Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n01 is online Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n02 is online Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_rsc_op_failure) warning: Unexpected result (not running) was recorded for monitor of drbd:0 on el8-a01n01 at Jul 14 21:55:46 2020 | rc=7 id=drbd_last_failure_0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n02 ] Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:0 has failed 1 times on el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:1 has failed 1 times on el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node1 (Started el8-a01n01) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node2 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Recover drbd:0 ( el8-a01n01 ) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave drbd:1 (Started el8-a01n02) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pcmk__log_transition_summary) notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-11.bz2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_te_invoke) info: Processing graph 2 (ref=pe_calc-dc-1594763746-33) derived from /var/lib/pacemaker/pengine/pe-input-11.bz2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n01 | action 4 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:7 exec-time:0ms queue-time:0ms Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=4:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=drbd_stop_0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/54) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=27, reply=1, notify=0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:drbd action:stop call_id:27 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit drbd named 'drbd' Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n01: Cancelled | call=25 key=drbd_monitor_60000 confirmed=true Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.47 2 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.48 (null) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=48 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=4:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;4:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763746, @last-run=1594763746, @exec-time=0 Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/54, version=0.13.48) Jul 14 21:55:46 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec_with_unit) debug: Calling StopUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 21:55:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1171 Jul 14 21:55:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=232ms, remaining=99768ms, start_delay=2000ms) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:27 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop is now complete (elapsed=2241ms, remaining=97759ms): not running (7) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:2241ms queue-time:0ms Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of stop operation for drbd on el8-a01n01: 0 (ok) | call=27 key=drbd_stop_0 confirmed=true cib-update=55 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/55) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.48 2 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.49 (null) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=49 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;4:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=27, @rc-code=0, @op-status=0, @exec-time=2241 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/55, version=0.13.49) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 2 action 4 (drbd_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=27 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating start operation drbd_start_0 locally on el8-a01n01 | action 20 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=20:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=drbd_start_0 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/56) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.49 2 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.50 (null) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=50 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=20:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;20:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763749, @last-run=1594763749, @exec-time=0 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=28, reply=1, notify=0 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:drbd action:start call_id:28 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit drbd named 'drbd' Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/56, version=0.13.50) Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec_with_unit) debug: Calling StartUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/1172 Jul 14 21:55:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd start may still be in progress: re-scheduling (elapsed=179ms, remaining=99821ms, start_delay=2000ms) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:28 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd start is now complete (elapsed=2187ms, remaining=97813ms): ok (0) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:28 exit-code:0 exec-time:2187ms queue-time:0ms Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of start operation for drbd on el8-a01n01: 0 (ok) | call=28 key=drbd_start_0 confirmed=true cib-update=57 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.50 2 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.51 (null) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=51 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;20:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=28, @rc-code=0, @op-status=0, @exec-time=2187 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.13.51) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 2 action 20 (drbd_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=28 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating monitor operation drbd_monitor_60000 locally on el8-a01n01 | action 3 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=3:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=drbd_monitor_60000 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=29, reply=1, notify=0 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:29 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.51 2 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.52 (null) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=52 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-key=3:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;3:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763751, @exec-time=0 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.13.52) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:29 exit-code:0 exec-time:4ms queue-time:0ms Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of monitor operation for drbd on el8-a01n01: 0 (ok) | call=29 key=drbd_monitor_60000 confirmed=false cib-update=59 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/59) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.52 2 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.53 (null) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=53 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;3:2:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=29, @rc-code=0, @op-status=0, @exec-time=4 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/59, version=0.13.53) Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 2 action 3 (drbd_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=29 Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (run_graph) notice: Transition 2 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-11.bz2): Complete Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 14 21:55:51 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 21:55:56 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 6a01ad05ce19e14ae684dfabd34d038a for 0.13.53 (0x563bd67251f0 0) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemakerd [6050] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemakerd [6050] (pcmk_shutdown_worker) notice: Shutting down Pacemaker Jul 14 21:56:33 el8-a01n01.digimer.ca pacemakerd [6050] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 6056 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_log) info: Input I_SHUTDOWN received in state S_IDLE from crm_shutdown Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n01) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_POLICY_ENGINE) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594763793 | from el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (write_attribute) info: Sent CIB request 6 with 1 change for shutdown (id n/a, set n/a) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/6) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.53 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.54 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=54 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/6, version=0.13.54) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (abort_transition_graph) info: Transition 2 aborted by status-1-shutdown doing create shutdown=1594763793: Transient attribute change | cib=0.13.54 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: CIB update 6 result for shutdown: OK | rc=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594763793 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n01 is shutting down Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status_fencing) info: Node el8-a01n02 is active Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n02 is online Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_rsc_op_failure) warning: Unexpected result (not running) was recorded for monitor of drbd:0 on el8-a01n01 at Jul 14 21:55:46 2020 | rc=7 id=drbd_last_failure_0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:0 has failed 1 times on el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:1 has failed 1 times on el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource hypervisor:0 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource drbd:0 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n02 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogNodeActions) notice: * Shutdown el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Move virsh_node1 ( el8-a01n01 -> el8-a01n02 ) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node2 (Started el8-a01n02) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave drbd:1 (Started el8-a01n02) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pcmk__log_transition_summary) notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-12.bz2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was member) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (handle_shutdown_request) info: Creating shutdown request for el8-a01n02 (state=S_POLICY_ENGINE) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594763793 | from el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (write_attribute) info: Sent CIB request 7 with 2 changes for shutdown (id n/a, set n/a) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/7) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_te_invoke) info: Processing graph 3 (ref=pe_calc-dc-1594763793-38) derived from /var/lib/pacemaker/pengine/pe-input-12.bz2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 locally on el8-a01n01 | action 7 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.54 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.55 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=55 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: ++ Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=7:3:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=virsh_node1_stop_0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=31, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:virsh_node1 action:stop call_id:31 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/attrd/7, version=0.13.55) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) info: finished - rsc:virsh_node1 action:stop call_id:31 exit-code:0 exec-time:1ms queue-time:0ms Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: CIB update 7 result for shutdown: OK | rc=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594763793 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-attrd [6054] (attrd_cib_callback) info: * shutdown[el8-a01n02]=1594763793 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (abort_transition_graph) notice: Transition 3 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.13.55 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=false Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) info: Result of monitor operation for virsh_node1 on el8-a01n01: Cancelled | call=22 key=virsh_node1_monitor_60000 confirmed=true Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.55 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.56 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=56 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=7:3:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;7:3:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok) | call=31 key=virsh_node1_stop_0 confirmed=true cib-update=62 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.13.56) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/62) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.56 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.57 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=57 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;7:3:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=31, @rc-code=0, @op-status=0, @exec-time=1 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 3 action 7 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=31 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (run_graph) notice: Transition 3 (Complete=3, Pending=0, Fired=0, Skipped=3, Incomplete=7, Source=/var/lib/pacemaker/pengine/pe-input-12.bz2): Stopped Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/62, version=0.13.57) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n01 is shutting down Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (determine_online_status) info: Node el8-a01n02 is shutting down Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_rsc_op_failure) warning: Unexpected result (not running) was recorded for monitor of drbd:0 on el8-a01n01 at Jul 14 21:55:46 2020 | rc=7 id=drbd_last_failure_0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 1 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (unpack_node_loop) info: Node 2 is already processed Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: hypervisor-clone [hypervisor] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (clone_print) info: Clone Set: drbd-clone [drbd] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ] Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:0 has failed 1 times on el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pe_get_failcount) info: drbd:1 has failed 1 times on el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (check_migration_threshold) info: drbd-clone can fail 999999 more times on el8-a01n01 before being forced off Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource virsh_node1 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource virsh_node2 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource hypervisor:0 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource hypervisor:1 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource drbd:0 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (native_color) info: Resource drbd:1 cannot run anywhere Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n02 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogNodeActions) notice: * Shutdown el8-a01n02 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogNodeActions) notice: * Shutdown el8-a01n01 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogActions) info: Leave virsh_node1 (Stopped) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop virsh_node2 ( el8-a01n02 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop hypervisor:1 ( el8-a01n02 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-schedulerd[6055] (pcmk__log_transition_summary) notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-13.bz2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_te_invoke) info: Processing graph 4 (ref=pe_calc-dc-1594763793-40) derived from /var/lib/pacemaker/pengine/pe-input-13.bz2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n02 | action 6 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n01 | action 7 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=7:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=hypervisor_stop_0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=33, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:hypervisor action:stop call_id:33 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit libvirtd named 'hypervisor' Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/64) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.57 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.58 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=58 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=7:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;7:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec_with_unit) debug: Calling StopUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/64, version=0.13.58) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.58 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.59 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=59 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=6:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;6:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/28, version=0.13.59) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.59 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.60 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=60 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;6:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=27, @rc-code=0, @op-status=0, @exec-time=1 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.13.60) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 on el8-a01n02 | action 8 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n01 | action 13 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:drbd action:monitor call_id:29 exit-code:0 exec-time:0ms queue-time:0ms Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=0, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (do_lrm_rsc_op) info: Performing key=13:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c op=drbd_stop_0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 29872d72-cf4e-4597-93c4-86f8b71374d4: rc=35, reply=1, notify=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) info: executing - rsc:drbd action:stop call_id:35 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit drbd named 'drbd' Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/65) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 14 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.60 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.61 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=61 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=13:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;13:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/65, version=0.13.61) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 4 action 6 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=27 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n01: Cancelled | call=24 key=hypervisor_monitor_60000 confirmed=true Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n01: Cancelled | call=29 key=drbd_monitor_60000 confirmed=true Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.61 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.62 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=62 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=8:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;8:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.13.62) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.62 2 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.63 (null) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=63 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=14:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @transition-magic=-1:193;14:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763793, @last-run=1594763793, @exec-time=0 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.13.63) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1267 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: hypervisor stop may still be in progress: re-scheduling (elapsed=300ms, remaining=99700ms, start_delay=2000ms) Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec_with_unit) debug: Calling StopUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1268 Jul 14 21:56:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=497ms, remaining=99503ms, start_delay=2000ms) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.63 2 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.64 (null) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=64 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;8:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2194, @queue-time=1 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.13.64) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 4 action 8 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=29 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:33 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor' Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: hypervisor stop is now complete (elapsed=2306ms, remaining=97694ms): not running (7) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:33 exit-code:0 exec-time:2307ms queue-time:0ms Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok) | call=33 key=hypervisor_stop_0 confirmed=true cib-update=66 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/66) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: --- 0.13.64 2 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: Diff: +++ 0.13.65 (null) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib: @num_updates=65 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;7:4:0:7fee6bf5-1c09-43c8-93d7-f6f42af0e11c, @call-id=33, @rc-code=0, @op-status=0, @exec-time=2307 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/66, version=0.13.65) Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-controld [6056] (process_graph_event) info: Transition 4 action 7 (hypervisor_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=33 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=2504ms, remaining=97496ms, start_delay=2000ms) Jul 14 21:56:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=4507ms, remaining=95493ms, start_delay=2000ms) Jul 14 21:56:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=6515ms, remaining=93485ms, start_delay=2000ms) Jul 14 21:56:40 el8-a01n01.digimer.ca pacemaker-based [6051] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 0265b0a70b084f185c477f8c2659b229 for 0.13.65 (0x563bd67251f0 0) Jul 14 21:56:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=8528ms, remaining=91472ms, start_delay=2000ms) Jul 14 21:56:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=10542ms, remaining=89458ms, start_delay=2000ms) Jul 14 21:56:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=12546ms, remaining=87454ms, start_delay=2000ms) Jul 14 21:56:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=14549ms, remaining=85451ms, start_delay=2000ms) Jul 14 21:56:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=16553ms, remaining=83447ms, start_delay=2000ms) Jul 14 21:56:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=18556ms, remaining=81444ms, start_delay=2000ms) Jul 14 21:56:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=20560ms, remaining=79440ms, start_delay=2000ms) Jul 14 21:56:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=22563ms, remaining=77437ms, start_delay=2000ms) Jul 14 21:56:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=24570ms, remaining=75430ms, start_delay=2000ms) Jul 14 21:56:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:56:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:56:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:56:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=26576ms, remaining=73424ms, start_delay=2000ms) Jul 14 21:57:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=28592ms, remaining=71408ms, start_delay=2000ms) Jul 14 21:57:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=30595ms, remaining=69405ms, start_delay=2000ms) Jul 14 21:57:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=32598ms, remaining=67402ms, start_delay=2000ms) Jul 14 21:57:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=34601ms, remaining=65399ms, start_delay=2000ms) Jul 14 21:57:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=36605ms, remaining=63395ms, start_delay=2000ms) Jul 14 21:57:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=38612ms, remaining=61388ms, start_delay=2000ms) Jul 14 21:57:13 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:13 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:13 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:13 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=40615ms, remaining=59385ms, start_delay=2000ms) Jul 14 21:57:15 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:15 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:15 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:15 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=42618ms, remaining=57382ms, start_delay=2000ms) Jul 14 21:57:17 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:17 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:17 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:17 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=44622ms, remaining=55378ms, start_delay=2000ms) Jul 14 21:57:19 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:19 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:19 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:19 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=46627ms, remaining=53373ms, start_delay=2000ms) Jul 14 21:57:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:21 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=48630ms, remaining=51370ms, start_delay=2000ms) Jul 14 21:57:23 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:23 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:23 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:23 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=50633ms, remaining=49367ms, start_delay=2000ms) Jul 14 21:57:25 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:25 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:25 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:25 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=52638ms, remaining=47362ms, start_delay=2000ms) Jul 14 21:57:27 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:27 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:27 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:27 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=54645ms, remaining=45355ms, start_delay=2000ms) Jul 14 21:57:29 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:29 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:29 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:29 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=56649ms, remaining=43351ms, start_delay=2000ms) Jul 14 21:57:31 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:31 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:31 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:31 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=58652ms, remaining=41348ms, start_delay=2000ms) Jul 14 21:57:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:33 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=60654ms, remaining=39346ms, start_delay=2000ms) Jul 14 21:57:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:35 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=62658ms, remaining=37342ms, start_delay=2000ms) Jul 14 21:57:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:37 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=64661ms, remaining=35339ms, start_delay=2000ms) Jul 14 21:57:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:39 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=66665ms, remaining=33335ms, start_delay=2000ms) Jul 14 21:57:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:41 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=68669ms, remaining=31331ms, start_delay=2000ms) Jul 14 21:57:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:43 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=70672ms, remaining=29328ms, start_delay=2000ms) Jul 14 21:57:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:45 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=72676ms, remaining=27324ms, start_delay=2000ms) Jul 14 21:57:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:47 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=74679ms, remaining=25321ms, start_delay=2000ms) Jul 14 21:57:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:49 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=76683ms, remaining=23317ms, start_delay=2000ms) Jul 14 21:57:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:51 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=78686ms, remaining=21314ms, start_delay=2000ms) Jul 14 21:57:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:53 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=80688ms, remaining=19312ms, start_delay=2000ms) Jul 14 21:57:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:55 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=82695ms, remaining=17305ms, start_delay=2000ms) Jul 14 21:57:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:57 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=84698ms, remaining=15302ms, start_delay=2000ms) Jul 14 21:57:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:57:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:57:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:57:59 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=86701ms, remaining=13299ms, start_delay=2000ms) Jul 14 21:58:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:01 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=88705ms, remaining=11295ms, start_delay=2000ms) Jul 14 21:58:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:03 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=90707ms, remaining=9293ms, start_delay=2000ms) Jul 14 21:58:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:05 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=92710ms, remaining=7290ms, start_delay=2000ms) Jul 14 21:58:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:07 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=94718ms, remaining=5282ms, start_delay=2000ms) Jul 14 21:58:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:09 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=96725ms, remaining=3275ms, start_delay=1637ms) Jul 14 21:58:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (log_execute) debug: executing - rsc:drbd action:monitor call_id:35 Jul 14 21:58:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd' Jul 14 21:58:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1 Jul 14 21:58:11 el8-a01n01.digimer.ca pacemaker-execd [6053] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=98371ms, remaining=1629ms, start_delay=814ms)