API tools faq
paste
digimer

Untitled

digimer
Jul 14th, 2020
156
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 258.16 KB | None | 0 0
raw download clone embed print report
  1. [root@el8-a01n01 ~]# cat /var/log/pacemaker/el8-a01n02/pacemaker.log.fence-fix-to-crash
  2. Jul 14 21:41:22 el8-a01n02.digimer.ca pacemakerd [1607] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  3. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (get_cluster_type) info: Detected an active 'corosync' cluster
  4. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (mcp_read_config) info: Reading configure for stack: corosync
  5. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (main) notice: Starting Pacemaker 2.0.3-5.el8_2.1 | build=4b1f869f0f features: generated-manpages agent-manpages ncurses libqb-logging libqb-ipc systemd nagios corosync-native atomic-attrd acls
  6. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (main) info: Maximum core file size is: 18446744073709551615
  7. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (qb_ipcs_us_publish) info: server name: pacemakerd
  8. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to lrmd IPC: Connection refused
  9. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to cib_ro IPC: Connection refused
  10. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to crmd IPC: Connection refused
  11. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection refused
  12. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to pengine IPC: Connection refused
  13. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk__ipc_is_authentic_process_active) info: Could not connect to stonith-ng IPC: Connection refused
  14. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Created entry 2f99bc49-a55c-4a79-b16e-32b7d11cb263/0x556a0c6228c0 for node el8-a01n02/2 (1 total)
  15. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  16. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Node 2 has uuid 2
  17. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n02[2] - corosync-cpg is now online
  18. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (cluster_connect_quorum) warning: Quorum lost
  19. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Created entry 2e33d8d4-d1c0-49f6-8a4f-bb05051c6739/0x556a0c624860 for node el8-a01n01/1 (2 total)
  20. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  21. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_get_peer) info: Node 1 has uuid 1
  22. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Using uid=189 and group=189 for process pacemaker-based
  23. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1608 for process pacemaker-based
  24. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1609 for process pacemaker-fenced
  25. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1610 for process pacemaker-execd
  26. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Using uid=189 and group=189 for process pacemaker-attrd
  27. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1611 for process pacemaker-attrd
  28. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Using uid=189 and group=189 for process pacemaker-schedulerd
  29. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1612 for process pacemaker-schedulerd
  30. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Using uid=189 and group=189 for process pacemaker-controld
  31. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (start_child) info: Forked child 1613 for process pacemaker-controld
  32. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (main) notice: Pacemaker daemon successfully started and accepting connections
  33. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_quorum_notification) info: Quorum still lost | membership=149 members=1
  34. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
  35. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_reap_unseen_nodes) info: State of node el8-a01n01[1] is still unknown
  36. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_cpg_membership) info: Group pacemakerd event 0: node 2 pid 1607 joined via cpg_join
  37. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_cpg_membership) info: Group pacemakerd event 0: el8-a01n02 (node 2 pid 1607) is member
  38. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  39. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (main) notice: Starting Pacemaker CIB manager
  40. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (get_cluster_type) info: Verifying cluster type: 'corosync'
  41. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (get_cluster_type) info: Assuming an active 'corosync' cluster
  42. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-execd [1610] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  43. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-execd [1610] (main) notice: Starting Pacemaker local executor
  44. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_ipcs_us_publish) info: server name: lrmd
  45. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-execd [1610] (main) notice: Pacemaker local executor successfully started and accepting connections
  46. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (retrieveCib) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
  47. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  48. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (main) notice: Starting Pacemaker scheduler
  49. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (qb_ipcs_us_publish) info: server name: pengine
  50. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (main) notice: Pacemaker scheduler successfully started and accepting connections
  51. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  52. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-attrd [1611] (main) notice: Starting Pacemaker node attribute manager
  53. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  54. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (main) notice: Starting Pacemaker fencer
  55. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (get_cluster_type) info: Verifying cluster type: 'corosync'
  56. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (get_cluster_type) info: Assuming an active 'corosync' cluster
  57. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  58. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  59. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-controld [1613] (main) notice: Starting Pacemaker controller
  60. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) info: Input I_STARTUP received in state S_STARTING from crmd_init
  61. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Created entry f4215d16-19e4-4e00-a4a5-e17516ec9fed/0x55722ccf7d60 for node el8-a01n02/2 (1 total)
  62. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  63. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  64. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Node 2 has uuid 2
  65. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n02[2] - corosync-cpg is now online
  66. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  67. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-fenced [1609] (init_cs_connection_once) info: Connection to 'corosync': established
  68. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Created entry 2721668f-57d5-4727-8aa4-82398ba591f6/0x55e7a1cffa70 for node el8-a01n02/2 (1 total)
  69. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  70. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Node 2 has uuid 2
  71. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n02[2] - corosync-cpg is now online
  72. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  73. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (init_cs_connection_once) info: Connection to 'corosync': established
  74. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (qb_ipcs_us_publish) info: server name: cib_ro
  75. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (qb_ipcs_us_publish) info: server name: cib_rw
  76. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (qb_ipcs_us_publish) info: server name: cib_shm
  77. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (main) notice: Pacemaker CIB manager successfully started and accepting connections
  78. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (pcmk_cpg_membership) info: Group cib event 0: node 2 pid 1608 joined via cpg_join
  79. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (pcmk_cpg_membership) info: Group cib event 0: el8-a01n02 (node 2 pid 1608) is member
  80. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-11.raw
  81. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Wrote version 0.11.0 of the CIB to disk (digest: f200e1165b560ade835f6f7bb132756a)
  82. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.LPwEa2 (digest: /var/lib/pacemaker/cib/cib.cAhDdE)
  83. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_cpg_membership) info: Group pacemakerd event 1: node 1 pid 26603 joined via cluster join
  84. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n01 (node 1 pid 26603) is member
  85. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n01[1] - corosync-cpg is now online
  86. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n02 (node 2 pid 1607) is member
  87. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining, last=7): Try again (6)
  88. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_quorum_notification) notice: Quorum acquired | membership=153 members=2
  89. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemakerd [1607] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
  90. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (pcmk_cpg_membership) info: Group cib event 1: node 1 pid 26604 joined via cluster join
  91. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Created entry e8ba2f58-5df3-4872-a190-f29ce7bd9e48/0x55e7a1d05070 for node el8-a01n01/1 (2 total)
  92. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  93. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_get_peer) info: Node 1 has uuid 1
  94. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n01 (node 1 pid 26604) is member
  95. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n01[1] - corosync-cpg is now online
  96. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  97. Jul 14 21:41:23 el8-a01n02.digimer.ca pacemaker-based [1608] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n02 (node 2 pid 1608) is member
  98. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (main) info: CIB connection active
  99. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (get_cluster_type) info: Verifying cluster type: 'corosync'
  100. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (get_cluster_type) info: Assuming an active 'corosync' cluster
  101. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  102. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Created entry 40d2d89e-a2ec-4663-a9fe-2280662c1064/0x55c961e8ca20 for node el8-a01n02/2 (1 total)
  103. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  104. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Node 2 has uuid 2
  105. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n02[2] - corosync-cpg is now online
  106. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  107. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (init_cs_connection_once) info: Connection to 'corosync': established
  108. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (get_cluster_type) info: Verifying cluster type: 'corosync'
  109. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (get_cluster_type) info: Assuming an active 'corosync' cluster
  110. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  111. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (main) info: Cluster connection active
  112. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_erase_attrs) info: Clearing transient attributes from CIB | xpath=//node_state[@uname='el8-a01n02']/transient_attributes
  113. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes to all (origin=local/attrd/2)
  114. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Created entry bb9747b7-bb11-4af9-85f5-0e2ed0e81762/0x561094a6be20 for node el8-a01n02/2 (1 total)
  115. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  116. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Cluster node el8-a01n02 is now in unknown state
  117. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (qb_ipcs_us_publish) info: server name: attrd
  118. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (main) notice: Pacemaker node attribute manager successfully started and accepting connections
  119. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (pcmk_cpg_membership) info: Group attrd event 0: node 2 pid 1611 joined via cpg_join
  120. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n02 (node 2 pid 1611) is member
  121. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2 | from el8-a01n02
  122. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_start_election_if_needed) info: Starting an election to determine the writer
  123. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Node 2 has uuid 2
  124. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n02[2] - corosync-cpg is now online
  125. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Node el8-a01n02 is now a peer | DC=<none> old=0x0000000 new=0x4000000
  126. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds
  127. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (init_cs_connection_once) info: Connection to 'corosync': established
  128. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (setup_cib) info: Watching for stonith topology changes
  129. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (qb_ipcs_us_publish) info: server name: stonith-ng
  130. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (main) notice: Pacemaker fencer successfully started and accepting connections
  131. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (election_check) info: election-attrd won by local node
  132. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset)
  133. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a
  134. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (pcmk_cpg_membership) info: Group stonith-ng event 0: node 2 pid 1609 joined via cpg_join
  135. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (pcmk_cpg_membership) info: Group stonith-ng event 0: el8-a01n02 (node 2 pid 1609) is member
  136. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 1 pid 26605 joined via cluster join
  137. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n02/attrd/2, version=0.11.0)
  138. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (cluster_connect_quorum) notice: Quorum acquired
  139. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Created entry 50fde642-79c9-47ac-a317-46d70313447d/0x55722cdff990 for node el8-a01n01/1 (2 total)
  140. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  141. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_get_peer) info: Node 1 has uuid 1
  142. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n01 (node 1 pid 26605) is member
  143. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n01[1] - corosync-cpg is now online
  144. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  145. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n02 (node 2 pid 1609) is member
  146. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (init_cib_cache_cb) info: Updating device list from the cib: init
  147. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (cib_devices_update) info: Updating devices to version 0.11.0
  148. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (pcmk_cpg_membership) info: Group attrd event 1: node 1 pid 26607 joined via cpg_join
  149. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Created entry b3950305-ae13-43dc-be89-58f694b9272d/0x561094a6e700 for node el8-a01n01/1 (2 total)
  150. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  151. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Cluster node el8-a01n01 is now in unknown state
  152. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_get_peer) info: Node 1 has uuid 1
  153. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Created entry 10e954fc-cb9b-42f2-a80b-041a64a947c8/0x55c961e8f520 for node el8-a01n01/1 (2 total)
  154. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  155. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_get_peer) info: Node 1 has uuid 1
  156. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n01 (node 1 pid 26607) is member
  157. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n01[1] - corosync-cpg is now online
  158. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  159. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (pcmk_cpg_membership) info: Group attrd event 1: el8-a01n02 (node 2 pid 1611) is member
  160. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_ha_control) info: Connected to the cluster
  161. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (lrmd_ipc_connect) info: Connecting to executor
  162. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (crm_client_new) debug: Connecting 0x55b3579dcfb0 for uid=189 gid=189 pid=1613 id=74d60a7f-ef31-4534-8c88-bc870fd67528
  163. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (handle_new_connection) debug: IPC credentials authenticated (/dev/shm/qb-1610-1613-9-c8LuQ6/qb)
  164. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_ipcs_shm_connect) debug: connecting to client [1613]
  165. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  166. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  167. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/4)
  168. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  169. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed register operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  170. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_control) info: Connection to the executor established
  171. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_started) info: Delaying start, no membership data (0000000000100000)
  172. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_started) info: Delaying start, no membership data (0000000000100000)
  173. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_quorum_notification) info: Quorum retained | membership=153 members=2
  174. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
  175. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Cluster node el8-a01n01 is now member (was in unknown state)
  176. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
  177. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Cluster node el8-a01n02 is now member (was in unknown state)
  178. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/4, version=0.11.0)
  179. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_started) info: Delaying start, Config not read (0000000000000040)
  180. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (qb_ipcs_us_publish) info: server name: crmd
  181. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_started) notice: Pacemaker controller successfully started and accepting connections
  182. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) info: Input I_PENDING received in state S_STARTING from do_started
  183. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started
  184. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes: OK (rc=0, origin=el8-a01n01/attrd/2, version=0.11.0)
  185. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2 | from el8-a01n01
  186. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a
  187. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/4, version=0.11.0)
  188. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 1615 exited with rc=0
  189. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (stonith_device_register) notice: Added 'virsh_node1' to the device list (1 active devices)
  190. Jul 14 21:41:24 el8-a01n02.digimer.ca pacemaker-fenced [1609] (stonith_device_register) notice: Added 'virsh_node2' to the device list (2 active devices)
  191. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_cpg_membership) info: Group crmd event 0: node 2 pid 1613 joined via cpg_join
  192. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n02 (node 2 pid 1613) is member
  193. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_cpg_membership) info: Group crmd event 1: node 1 pid 26609 joined via cpg_join
  194. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n01 (node 1 pid 26609) is member
  195. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n01[1] - corosync-cpg is now online
  196. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (peer_update_callback) info: Node el8-a01n01 is now a peer | DC=<none> old=0x0000000 new=0x4000000
  197. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds
  198. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (pcmk_cpg_membership) info: Group crmd event 1: el8-a01n02 (node 2 pid 1613) is member
  199. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 30 seconds
  200. Jul 14 21:41:25 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_connect_stonith) notice: Fencer successfully connected
  201. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_timer_popped) info: Election Trigger just popped | input=I_DC_TIMEOUT time=20000ms
  202. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) warning: Input I_DC_TIMEOUT received in state S_PENDING from crm_timer_popped
  203. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_PENDING -> S_ELECTION | input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped
  204. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (election_check) info: election-DC won by local node
  205. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb
  206. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb
  207. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_control) info: Registering TE UUID: 11323fbf-740e-4c7f-8ee5-4d98173fee16
  208. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (set_graph_functions) info: Setting custom graph functions
  209. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_dc_takeover) info: Taking over DC status for this partition
  210. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_readwrite) info: We are now in R/W mode
  211. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.11.0)
  212. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/8)
  213. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/10)
  214. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/12)
  215. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (join_make_offer) info: Making join-1 offers based on membership event 153
  216. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (join_make_offer) info: Sending join-1 offer to el8-a01n01
  217. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (join_make_offer) info: Sending join-1 offer to el8-a01n02
  218. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_dc_join_offer_all) info: Waiting on join-1 requests from 2 outstanding nodes
  219. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (update_dc) info: Set DC to el8-a01n02 (3.3.0)
  220. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_expected) info: update_dc: Node el8-a01n02[2] - expected state is now member (was (null))
  221. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n02/crmd/8, version=0.11.0)
  222. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n02/crmd/10, version=0.11.0)
  223. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n02/crmd/12, version=0.11.0)
  224. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/14)
  225. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n02/crmd/14, version=0.11.0)
  226. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_expected) info: do_dc_join_filter_offer: Node el8-a01n01[1] - expected state is now member (was (null))
  227. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
  228. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/18)
  229. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_replace) info: Digest matched on replace from el8-a01n02: 58002fb42d021891f86a2c77da32294a
  230. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_replace) info: Replaced 0.11.0 with 0.11.0 from el8-a01n02
  231. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (controld_delete_node_state) info: Deleting resource history for node el8-a01n01 (via CIB call 20) | xpath=//node_state[@uname='el8-a01n01']/lrm
  232. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (controld_delete_node_state) info: Deleting resource history for node el8-a01n02 (via CIB call 22) | xpath=//node_state[@uname='el8-a01n02']/lrm
  233. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=el8-a01n02/crmd/17, version=0.11.0)
  234. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/18, version=0.11.0)
  235. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/19)
  236. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm to all (origin=local/crmd/20)
  237. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/21)
  238. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm to all (origin=local/crmd/22)
  239. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/23)
  240. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/19, version=0.11.0)
  241. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm: OK (rc=0, origin=el8-a01n02/crmd/20, version=0.11.0)
  242. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.0 2
  243. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.1 (null)
  244. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=1
  245. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status: <node_state id="1" uname="el8-a01n01" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
  246. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm id="1">
  247. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_resources/>
  248. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm>
  249. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </node_state>
  250. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/21, version=0.11.1)
  251. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm: OK (rc=0, origin=el8-a01n02/crmd/22, version=0.11.1)
  252. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.1 2
  253. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.2 (null)
  254. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=2
  255. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status: <node_state id="2" uname="el8-a01n02" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
  256. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm id="2">
  257. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_resources/>
  258. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm>
  259. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </node_state>
  260. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
  261. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/23, version=0.11.2)
  262. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true
  263. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/26)
  264. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27)
  265. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/28)
  266. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/26, version=0.11.2)
  267. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.2 2
  268. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.3 (null)
  269. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=3
  270. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition, @join=member, @expected=member
  271. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition, @join=member, @expected=member
  272. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/27, version=0.11.3)
  273. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.3 2
  274. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.4 (null)
  275. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=4, @dc-uuid=2
  276. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n02/crmd/28, version=0.11.4)
  277. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-12.raw
  278. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  279. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  280. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  281. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  282. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  283. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  284. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  285. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  286. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  287. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  288. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  289. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  290. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  291. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  292. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  293. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  294. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:0 on el8-a01n01
  295. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:1 on el8-a01n02
  296. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01
  297. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n02
  298. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Wrote version 0.11.0 of the CIB to disk (digest: 9a696706e038f86744f31e5794c481e4)
  299. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start virsh_node1 ( el8-a01n01 )
  300. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start virsh_node2 ( el8-a01n02 )
  301. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start hypervisor:0 ( el8-a01n01 )
  302. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start hypervisor:1 ( el8-a01n02 )
  303. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start drbd:0 ( el8-a01n01 )
  304. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start drbd:1 ( el8-a01n02 )
  305. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AtyvVY (digest: /var/lib/pacemaker/cib/cib.UeJnmE)
  306. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-0.bz2
  307. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  308. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 0 (ref=pe_calc-dc-1594762905-9) derived from /var/lib/pacemaker/pengine/pe-input-0.bz2
  309. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n02 | action 5
  310. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node1' not in cache
  311. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  312. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node1'
  313. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=1
  314. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  315. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=5:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node1_monitor_0
  316. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 1619 exited with rc=0
  317. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=5, reply=1, notify=0
  318. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/30)
  319. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:5
  320. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (stonith_api_signon) debug: Attempting fencer connection by pacemaker-execd with mainloop
  321. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n01 | action 1
  322. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  323. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  324. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  325. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (stonith_api_signon) debug: Connection to fencer by pacemaker-execd succeeded (registration token: a5c05a28-71c5-4a18-a94c-6e3fd30f178e)
  326. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.4 2
  327. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.5 (null)
  328. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=5
  329. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource
  330. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
  331. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="5:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;5:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-statu
  332. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  333. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:5 exit-code:7 exec-time:3ms queue-time:0ms
  334. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n02 | action 6
  335. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node2' not in cache
  336. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  337. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.11.5)
  338. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node2'
  339. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=1
  340. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  341. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=6:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_monitor_0
  342. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=9, reply=1, notify=0
  343. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31)
  344. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:9
  345. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:9 exit-code:7 exec-time:0ms queue-time:0ms
  346. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n01 | action 2
  347. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.5 2
  348. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of probe operation for virsh_node1 on el8-a01n02: 7 (not running) | call=5 key=virsh_node1_monitor_0 confirmed=true cib-update=32
  349. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.6 (null)
  350. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of probe operation for virsh_node2 on el8-a01n02: 7 (not running) | call=9 key=virsh_node2_monitor_0 confirmed=true cib-update=33
  351. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=6
  352. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
  353. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;6:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-statu
  354. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  355. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.11.6)
  356. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/32)
  357. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/33)
  358. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.6 2
  359. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.7 (null)
  360. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=7
  361. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;5:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=5, @rc-code=7, @op-status=0, @exec-time=3
  362. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.11.7)
  363. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.7 2
  364. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.8 (null)
  365. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=8
  366. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;6:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=9, @rc-code=7, @op-status=0
  367. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 5 (virsh_node1_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=5
  368. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.11.8)
  369. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_0 locally on el8-a01n02 | action 7
  370. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor' not in cache
  371. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  372. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor:1' not in cache
  373. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  374. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_rsc_register) info: Cached agent information for 'hypervisor'
  375. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=1
  376. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  377. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=7:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=hypervisor_monitor_0
  378. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=14, reply=1, notify=0
  379. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/34)
  380. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:14
  381. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.8 2
  382. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.9 (null)
  383. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=9
  384. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
  385. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="7:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;7:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-status=
  386. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  387. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.11.9)
  388. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  389. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 6 (virsh_node2_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=9
  390. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_0 locally on el8-a01n02 | action 8
  391. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'drbd' not in cache
  392. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  393. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_get_rsc_info) info: Agent information for 'drbd:1' not in cache
  394. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  395. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_rsc_register) info: Cached agent information for 'drbd'
  396. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=1
  397. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  398. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=8:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=drbd_monitor_0
  399. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/35)
  400. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=19, reply=1, notify=0
  401. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:19
  402. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  403. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.9 2
  404. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.10 (null)
  405. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=10
  406. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="drbd" type="drbd" class="systemd"/>
  407. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="8:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;8:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-status="-1" interva
  408. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  409. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/35, version=0.11.10)
  410. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  411. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  412. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:14 exit-code:7 exec-time:58ms queue-time:0ms
  413. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of probe operation for hypervisor on el8-a01n02: 7 (not running) | call=14 key=hypervisor_monitor_0 confirmed=true cib-update=36
  414. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
  415. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.10 2
  416. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.11 (null)
  417. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=11
  418. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;7:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=14, @rc-code=7, @op-status=0, @exec-time=58
  419. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/36, version=0.11.11)
  420. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 7 (hypervisor_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=14
  421. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:19 exit-code:7 exec-time:59ms queue-time:0ms
  422. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of probe operation for drbd on el8-a01n02: 7 (not running) | call=19 key=drbd_monitor_0 confirmed=true cib-update=37
  423. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/37)
  424. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.11 2
  425. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.12 (null)
  426. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=12
  427. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;8:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=19, @rc-code=7, @op-status=0, @exec-time=59
  428. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/37, version=0.11.12)
  429. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 8 (drbd_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=19
  430. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.12 2
  431. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.13 (null)
  432. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=13
  433. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
  434. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
  435. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="1:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;1:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-statu
  436. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  437. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/8, version=0.11.13)
  438. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.13 2
  439. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.14 (null)
  440. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=14
  441. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;1:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=5, @rc-code=7, @op-status=0, @exec-time=5
  442. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 1 (virsh_node1_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=5
  443. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 on el8-a01n01 | action 9
  444. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/9, version=0.11.14)
  445. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.14 2
  446. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.15 (null)
  447. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=15
  448. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
  449. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="2:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;2:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-statu
  450. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  451. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/10, version=0.11.15)
  452. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.15 2
  453. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.16 (null)
  454. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=16
  455. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;2:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=9, @rc-code=7, @op-status=0
  456. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/11, version=0.11.16)
  457. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 2 (virsh_node2_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=9
  458. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n02 | action 11
  459. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_start_0
  460. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=20, reply=1, notify=0
  461. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:virsh_node2 action:start call_id:20
  462. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_0 on el8-a01n01 | action 3
  463. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.16 2
  464. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.17 (null)
  465. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=17
  466. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  467. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/12, version=0.11.17)
  468. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/38)
  469. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.17 2
  470. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.18 (null)
  471. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=18
  472. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
  473. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;3:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-status=
  474. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  475. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/13, version=0.11.18)
  476. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.18 2
  477. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.19 (null)
  478. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=19
  479. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1
  480. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/38, version=0.11.19)
  481. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.19 2
  482. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.20 (null)
  483. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=20
  484. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;3:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=15, @rc-code=7, @op-status=0, @exec-time=88
  485. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 3 (hypervisor_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=15
  486. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_0 on el8-a01n01 | action 4
  487. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation hypervisor:1_start_0 locally on el8-a01n02 | action 15
  488. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=15:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=hypervisor_start_0
  489. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=21, reply=1, notify=0
  490. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/14, version=0.11.20)
  491. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:hypervisor action:start call_id:21
  492. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit libvirtd named 'hypervisor'
  493. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/39)
  494. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.20 2
  495. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.21 (null)
  496. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=21
  497. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="drbd" type="drbd" class="systemd"/>
  498. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="4:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;4:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-status="-1" interva
  499. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </lrm_resource>
  500. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/15, version=0.11.21)
  501. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.21 2
  502. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.22 (null)
  503. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=22
  504. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=15:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;15:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  505. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/39, version=0.11.22)
  506. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.22 2
  507. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.23 (null)
  508. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=23
  509. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;4:0:7:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=20, @rc-code=7, @op-status=0, @exec-time=20
  510. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/16, version=0.11.23)
  511. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 4 (drbd_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=20
  512. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation hypervisor:0_start_0 on el8-a01n01 | action 13
  513. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.23 2
  514. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.24 (null)
  515. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=24
  516. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=13:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;13:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  517. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/17, version=0.11.24)
  518. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec_with_unit) debug: Calling StartUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
  519. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/942
  520. Jul 14 21:41:45 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: hypervisor start may still be in progress: re-scheduling (elapsed=248ms, remaining=99752ms, start_delay=2000ms)
  521. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:21
  522. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  523. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  524. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: hypervisor start is now complete (elapsed=2253ms, remaining=97747ms): ok (0)
  525. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:21 exit-code:0 exec-time:2253ms queue-time:0ms
  526. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of start operation for hypervisor on el8-a01n02: 0 (ok) | call=21 key=hypervisor_start_0 confirmed=true cib-update=40
  527. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/40)
  528. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.24 2
  529. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.25 (null)
  530. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=25
  531. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;15:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=21, @rc-code=0, @op-status=0, @exec-time=2253
  532. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/40, version=0.11.25)
  533. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 15 (hypervisor_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=21
  534. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_60000 locally on el8-a01n02 | action 16
  535. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=16:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=hypervisor_monitor_60000
  536. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/41)
  537. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.25 2
  538. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.26 (null)
  539. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=26
  540. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']: <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="16:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;16:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01
  541. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/41, version=0.11.26)
  542. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=22, reply=1, notify=0
  543. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:22
  544. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  545. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  546. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:2ms queue-time:1ms
  547. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of monitor operation for hypervisor on el8-a01n02: 0 (ok) | call=22 key=hypervisor_monitor_60000 confirmed=false cib-update=42
  548. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/42)
  549. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.26 2
  550. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.27 (null)
  551. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=27
  552. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']: @transition-magic=0:0;16:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=22, @rc-code=0, @op-status=0, @exec-time=2, @queue-time=1
  553. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 16 (hypervisor_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=22
  554. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation drbd:1_start_0 locally on el8-a01n02 | action 23
  555. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=23:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=drbd_start_0
  556. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/42, version=0.11.27)
  557. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/43)
  558. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.27 2
  559. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.28 (null)
  560. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=23, reply=1, notify=0
  561. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=28
  562. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=23:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;23:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762907, @last-run=1594762907, @exec-time=0
  563. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:drbd action:start call_id:23
  564. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit drbd named 'drbd'
  565. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/43, version=0.11.28)
  566. Jul 14 21:41:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec_with_unit) debug: Calling StartUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice
  567. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.28 2
  568. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.29 (null)
  569. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=29
  570. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;13:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=21, @rc-code=0, @op-status=0, @exec-time=2364, @queue-time=1
  571. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/18, version=0.11.29)
  572. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 13 (hypervisor_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=21
  573. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_60000 on el8-a01n01 | action 14
  574. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.29 2
  575. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.30 (null)
  576. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=30
  577. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']: <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="14:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;14:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01
  578. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/19, version=0.11.30)
  579. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.30 2
  580. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.31 (null)
  581. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=31
  582. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']: @transition-magic=0:0;14:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=22, @rc-code=0, @op-status=0, @exec-time=5
  583. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/20, version=0.11.31)
  584. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 14 (hypervisor_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=22
  585. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation drbd:0_start_0 on el8-a01n01 | action 21
  586. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.31 2
  587. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.32 (null)
  588. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=32
  589. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=21:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;21:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762908, @last-run=1594762908, @exec-time=0
  590. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/21, version=0.11.32)
  591. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/1026
  592. Jul 14 21:41:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd start may still be in progress: re-scheduling (elapsed=186ms, remaining=99814ms, start_delay=2000ms)
  593. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:23
  594. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  595. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  596. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd start is now complete (elapsed=2191ms, remaining=97809ms): ok (0)
  597. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:23 exit-code:0 exec-time:2191ms queue-time:0ms
  598. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of start operation for drbd on el8-a01n02: 0 (ok) | call=23 key=drbd_start_0 confirmed=true cib-update=44
  599. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/44)
  600. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.32 2
  601. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.33 (null)
  602. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=33
  603. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;23:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=23, @rc-code=0, @op-status=0, @exec-time=2191
  604. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/44, version=0.11.33)
  605. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 23 (drbd_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=23
  606. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_60000 locally on el8-a01n02 | action 24
  607. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=24:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=drbd_monitor_60000
  608. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/45)
  609. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=24, reply=1, notify=0
  610. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:24
  611. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.33 2
  612. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.34 (null)
  613. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=34
  614. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  615. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="24:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;24:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" call-id="-1"
  616. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/45, version=0.11.34)
  617. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  618. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:4ms queue-time:0ms
  619. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of monitor operation for drbd on el8-a01n02: 0 (ok) | call=24 key=drbd_monitor_60000 confirmed=false cib-update=46
  620. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/46)
  621. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.34 2
  622. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.35 (null)
  623. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=35
  624. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;24:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=24, @rc-code=0, @op-status=0, @exec-time=4
  625. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/46, version=0.11.35)
  626. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 24 (drbd_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=24
  627. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.35 2
  628. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.36 (null)
  629. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=36
  630. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;21:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=23, @rc-code=0, @op-status=0, @exec-time=2216
  631. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/22, version=0.11.36)
  632. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 21 (drbd_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=23
  633. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_60000 on el8-a01n01 | action 22
  634. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.36 2
  635. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.37 (null)
  636. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=37
  637. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="22:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;22:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call-id="-1"
  638. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/23, version=0.11.37)
  639. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.37 2
  640. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.11.38 (null)
  641. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=38
  642. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;22:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=24, @rc-code=0, @op-status=0, @exec-time=5
  643. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/24, version=0.11.38)
  644. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 0 action 22 (drbd_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=24
  645. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_1:1621:stderr [ 2020-07-14 21:41:50,807 ERROR: Unable to connect/login to fencing device ]
  646. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_1:1621:stderr [ ]
  647. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_1:1621:stderr [ ]
  648. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1621] stderr: [ 2020-07-14 21:41:50,807 ERROR: Unable to connect/login to fencing device ]
  649. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1621] stderr: [ ]
  650. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1621] stderr: [ ]
  651. Jul 14 21:41:50 el8-a01n02.digimer.ca pacemaker-fenced [1609] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (monitor). remaining timeout is 15
  652. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_procfs_pid_of) info: Found pacemaker-based active as process 1608
  653. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-controld [1613] (throttle_send_command) info: New throttle mode: negligible load (was undetermined)
  654. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.11.38 2
  655. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.0 (null)
  656. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @epoch=12, @num_updates=0
  657. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='virsh_node1']/instance_attributes[@id='virsh_node1-instance_attributes']/nvpair[@id='virsh_node1-instance_attributes-ipaddr']: @value=192.168.122.1
  658. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.12.0)
  659. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-fenced [1609] (update_cib_stonith_devices_v2) info: Updating device list from the cib: modify nvpair[@id='virsh_node1-instance_attributes-ipaddr']
  660. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-fenced [1609] (cib_devices_update) info: Updating devices to version 0.12.0
  661. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 0 aborted by virsh_node1-instance_attributes-ipaddr doing modify ipaddr=192.168.122.1: Configuration change | cib=0.12.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='virsh_node1']/instance_attributes[@id='virsh_node1-instance_attributes']/nvpair[@id='virsh_node1-instance_attributes-ipaddr'] complete=false
  662. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-fenced [1609] (stonith_device_register) notice: Added 'virsh_node1' to the device list (2 active devices)
  663. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-13.raw
  664. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Wrote version 0.12.0 of the CIB to disk (digest: daf4c048d1270debdb32bf77a2ca60d5)
  665. Jul 14 21:41:54 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.1isKcr (digest: /var/lib/pacemaker/cib/cib.lLeUUy)
  666. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_2:1715:stderr [ 2020-07-14 21:41:57,031 ERROR: Unable to connect/login to fencing device ]
  667. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_2:1715:stderr [ ]
  668. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (operation_finished) notice: fence_virsh_monitor_2:1715:stderr [ ]
  669. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1715] stderr: [ 2020-07-14 21:41:57,031 ERROR: Unable to connect/login to fencing device ]
  670. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1715] stderr: [ ]
  671. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_action) warning: fence_virsh[1715] stderr: [ ]
  672. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (monitor) the maximum number of times (2) allowed
  673. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-fenced [1609] (log_operation) notice: Operation 'monitor' [1715] for device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  674. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) info: finished - rsc:virsh_node2 action:start call_id:20 exit-code:1 exec-time:11535ms queue-time:0ms
  675. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of start operation for virsh_node2 on el8-a01n02: 1 (error) | call=20 key=virsh_node2_start_0 confirmed=true cib-update=47
  676. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/47)
  677. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.0 2
  678. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.1 (null)
  679. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=1
  680. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:1;11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=20, @rc-code=1, @op-status=0, @exec-time=11535
  681. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']: <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="0:1;11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n02" ca
  682. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/47, version=0.12.1)
  683. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by operation virsh_node2_start_0 'modify' on el8-a01n02: Event failed | magic=0:1;11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 cib=0.12.1 source=process_graph_event:453 complete=false
  684. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (update_failcount) info: Updating failcount for virsh_node2 on el8-a01n02 after failed start: rc=1 (update=INFINITY, time=1594762917)
  685. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) notice: Transition 0 action 11 (virsh_node2_start_0 on el8-a01n02): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=20
  686. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n02
  687. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 4 with 1 change for fail-count-virsh_node2#start_0 (id n/a, set n/a)
  688. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/4)
  689. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n02]: (unset) -> 1594762917 | from el8-a01n02
  690. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 5 with 1 change for last-failure-virsh_node2#start_0 (id n/a, set n/a)
  691. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.1 2
  692. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.2 (null)
  693. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=2
  694. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/>
  695. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <instance_attributes id="status-2">
  696. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <nvpair id="status-2-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
  697. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </instance_attributes>
  698. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </transient_attributes>
  699. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/4, version=0.12.2)
  700. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 4 result for fail-count-virsh_node2#start_0: OK | rc=0
  701. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n02]=INFINITY
  702. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/5)
  703. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.12.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=false
  704. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.2 2
  705. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.3 (null)
  706. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=3
  707. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762917"/>
  708. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.12.3)
  709. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 5 result for last-failure-virsh_node2#start_0: OK | rc=0
  710. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n02]=1594762917
  711. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by status-2-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762917: Transient attribute change | cib=0.12.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false
  712. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.3 2
  713. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.4 (null)
  714. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=4
  715. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=10, @rc-code=1, @op-status=0, @exec-time=11666
  716. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']: <lrm_rsc_op id="virsh_node1_last_failure_0" operation_key="virsh_node1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call
  717. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/25, version=0.12.4)
  718. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by operation virsh_node1_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 cib=0.12.4 source=process_graph_event:453 complete=false
  719. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (update_failcount) info: Updating failcount for virsh_node1 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762917)
  720. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) notice: Transition 0 action 9 (virsh_node1_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=10
  721. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (run_graph) notice: Transition 0 (Complete=22, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-0.bz2): Complete
  722. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  723. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node1#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n02
  724. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 6 with 1 change for fail-count-virsh_node1#start_0 (id n/a, set n/a)
  725. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node1#start_0[el8-a01n01]: (unset) -> 1594762917 | from el8-a01n02
  726. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 7 with 1 change for last-failure-virsh_node1#start_0 (id n/a, set n/a)
  727. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/6)
  728. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/7)
  729. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.4 2
  730. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.5 (null)
  731. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=5
  732. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
  733. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <instance_attributes id="status-1">
  734. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ <nvpair id="status-1-fail-count-virsh_node1.start_0" name="fail-count-virsh_node1#start_0" value="INFINITY"/>
  735. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </instance_attributes>
  736. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ </transient_attributes>
  737. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.12.5)
  738. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 6 result for fail-count-virsh_node1#start_0: OK | rc=0
  739. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node1#start_0[el8-a01n01]=INFINITY
  740. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.5 2
  741. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.6 (null)
  742. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=6
  743. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-virsh_node1.start_0" name="last-failure-virsh_node1#start_0" value="1594762917"/>
  744. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.12.6)
  745. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 7 result for last-failure-virsh_node1#start_0: OK | rc=0
  746. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node1#start_0[el8-a01n01]=1594762917
  747. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.12.5 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=true
  748. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 0 aborted by status-1-last-failure-virsh_node1.start_0 doing create last-failure-virsh_node1#start_0=1594762917: Transient attribute change | cib=0.12.6 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  749. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  750. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  751. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  752. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  753. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node1_start_0 on el8-a01n01 changed: was 2c1306c93f0b4781ffe561961fcbb917 vs. now 5954d092e67eaaaf96b50bb5a144e5ff (reload:3.3.0) 0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  754. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node1 on el8-a01n01 because resource parameters have changed | virsh_node1_clear_failcount_0
  755. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node1_last_failure_0
  756. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node1_last_0
  757. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_failure_0
  758. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_0
  759. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  760. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  761. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  762. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  763. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n01
  764. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n02
  765. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  766. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  767. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  768. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  769. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  770. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  771. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node1_start_0 on el8-a01n01 changed: was 2c1306c93f0b4781ffe561961fcbb917 vs. now 5954d092e67eaaaf96b50bb5a144e5ff (reload:3.3.0) 0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  772. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_action_definition) info: params:reload <parameters login="root" port="el8-a01n01" pcmk_host_list="el8-a01n01" delay="15" password_script="/root/stonith.pw" ipaddr="192.168.122.1"/>
  773. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  774. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  775. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Recover virsh_node1 ( el8-a01n01 )
  776. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Recover virsh_node2 ( el8-a01n02 -> el8-a01n01 )
  777. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  778. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  779. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  780. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  781. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-1.bz2
  782. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (handle_response) info: pe_calc calculation pe_calc-dc-1594762917-29 is obsolete
  783. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  784. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  785. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  786. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  787. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node1_start_0 on el8-a01n01 changed: was 2c1306c93f0b4781ffe561961fcbb917 vs. now 5954d092e67eaaaf96b50bb5a144e5ff (reload:3.3.0) 0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  788. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node1 on el8-a01n01 because resource parameters have changed | virsh_node1_clear_failcount_0
  789. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node1_last_failure_0
  790. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node1_last_0
  791. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_failure_0
  792. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_0
  793. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  794. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  795. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  796. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  797. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n01
  798. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n02
  799. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  800. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  801. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  802. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  803. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  804. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  805. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node1_start_0 on el8-a01n01 changed: was 2c1306c93f0b4781ffe561961fcbb917 vs. now 5954d092e67eaaaf96b50bb5a144e5ff (reload:3.3.0) 0:1;9:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  806. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_action_definition) info: params:reload <parameters login="root" port="el8-a01n01" pcmk_host_list="el8-a01n01" delay="15" password_script="/root/stonith.pw" ipaddr="192.168.122.1"/>
  807. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  808. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node1 on el8-a01n01 because action definition changed | virsh_node1_clear_failcount_0
  809. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  810. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  811. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Recover virsh_node1 ( el8-a01n01 )
  812. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Recover virsh_node2 ( el8-a01n02 -> el8-a01n01 )
  813. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  814. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  815. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  816. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  817. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2
  818. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  819. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 2 (ref=pe_calc-dc-1594762917-30) derived from /var/lib/pacemaker/pengine/pe-input-2.bz2
  820. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 on el8-a01n01 | action 2
  821. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_crm_command) info: Executing crm-event (1) without waiting: clear_failcount on el8-a01n01
  822. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n02 | action 5
  823. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=5:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_stop_0
  824. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=25, reply=1, notify=0
  825. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:25
  826. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/51)
  827. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:25 exit-code:0 exec-time:1ms queue-time:0ms
  828. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.6 2
  829. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.7 (null)
  830. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=7
  831. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=2:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;2:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762917, @last-run=1594762917, @exec-time=0
  832. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/26, version=0.12.7)
  833. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.7 2
  834. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.8 (null)
  835. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=8
  836. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=5:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;5:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762917, @last-run=1594762917, @exec-time=0
  837. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/51, version=0.12.8)
  838. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.8 2
  839. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.9 (null)
  840. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=9
  841. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node1#start_0[el8-a01n01]: 1594762917 -> (unset) | from el8-a01n01
  842. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 8 with 1 change for last-failure-virsh_node1#start_0 (id n/a, set n/a)
  843. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node1#start_0[el8-a01n01]: INFINITY -> (unset) | from el8-a01n01
  844. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 9 with 1 change for fail-count-virsh_node1#start_0 (id n/a, set n/a)
  845. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;2:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=25, @rc-code=0, @op-status=0, @exec-time=1
  846. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/27, version=0.12.9)
  847. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.9 2
  848. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.10 (null)
  849. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_failure_0']
  850. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=10
  851. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='el8-a01n01']/lrm/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_failure_0']: OK (rc=0, origin=el8-a01n01/crmd/28, version=0.12.10)
  852. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/8)
  853. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/9)
  854. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.10 2
  855. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.11 (null)
  856. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-virsh_node1.start_0']
  857. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=11
  858. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/8, version=0.12.11)
  859. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 8 result for last-failure-virsh_node1#start_0: OK | rc=0
  860. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node1#start_0[el8-a01n01]=(null)
  861. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.11 2
  862. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.12 (null)
  863. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-virsh_node1.start_0']
  864. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=12
  865. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/9, version=0.12.12)
  866. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 9 result for fail-count-virsh_node1#start_0: OK | rc=0
  867. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node1#start_0[el8-a01n01]=(null)
  868. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n02: 0 (ok) | call=25 key=virsh_node2_stop_0 confirmed=true cib-update=52
  869. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52)
  870. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 2 action 2 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=25
  871. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.12 2
  872. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.13 (null)
  873. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=13
  874. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;5:2:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=25, @rc-code=0, @op-status=0, @exec-time=1
  875. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/52, version=0.12.13)
  876. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 2 aborted by deletion of lrm_rsc_op[@id='virsh_node1_last_failure_0']: Resource operation removal | cib=0.12.10 source=process_op_deletion:374 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_failure_0'] complete=false
  877. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 2 aborted by deletion of nvpair[@id='status-1-last-failure-virsh_node1.start_0']: Transient attribute change | cib=0.12.11 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-virsh_node1.start_0'] complete=false
  878. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 2 aborted by deletion of nvpair[@id='status-1-fail-count-virsh_node1.start_0']: Transient attribute change | cib=0.12.12 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-virsh_node1.start_0'] complete=false
  879. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 2 action 5 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=25
  880. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (run_graph) notice: Transition 2 (Complete=3, Pending=0, Fired=0, Skipped=2, Incomplete=4, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Stopped
  881. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  882. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  883. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  884. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  885. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  886. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_failure_0
  887. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  888. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  889. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  890. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  891. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  892. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  893. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  894. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  895. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  896. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  897. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  898. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  899. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  900. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  901. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start virsh_node1 ( el8-a01n01 )
  902. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start virsh_node2 ( el8-a01n01 )
  903. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  904. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  905. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  906. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  907. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-3.bz2
  908. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  909. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 3 (ref=pe_calc-dc-1594762917-34) derived from /var/lib/pacemaker/pengine/pe-input-3.bz2
  910. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 on el8-a01n01 | action 5
  911. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 on el8-a01n01 | action 7
  912. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.13 2
  913. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.14 (null)
  914. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=14
  915. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=5:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;5:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @op-digest=5954d092e67eaaaf96b50bb5a144e5ff, @op-s
  916. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/29, version=0.12.14)
  917. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.14 2
  918. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.15 (null)
  919. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=15
  920. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762917, @last-run=1594762917
  921. Jul 14 21:41:57 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/30, version=0.12.15)
  922. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.15 2
  923. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.16 (null)
  924. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=16
  925. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;5:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=26, @rc-code=0, @op-status=0, @exec-time=668
  926. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/31, version=0.12.16)
  927. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 3 action 5 (virsh_node1_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=26
  928. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_60000 on el8-a01n01 | action 6
  929. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.16 2
  930. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.17 (null)
  931. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=17
  932. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']: <lrm_rsc_op id="virsh_node1_monitor_60000" operation_key="virsh_node1_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;6:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a0
  933. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/32, version=0.12.17)
  934. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.17 2
  935. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.12.18 (null)
  936. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=18
  937. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_monitor_60000']: @transition-magic=0:0;6:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=28, @rc-code=0, @op-status=0, @exec-time=675
  938. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/33, version=0.12.18)
  939. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 3 action 6 (virsh_node1_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=28
  940. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.12.18 2
  941. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.0 (null)
  942. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @epoch=13, @num_updates=0
  943. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='virsh_node2']/instance_attributes[@id='virsh_node2-instance_attributes']/nvpair[@id='virsh_node2-instance_attributes-ipaddr']: @value=192.168.122.1
  944. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-fenced [1609] (update_cib_stonith_devices_v2) info: Updating device list from the cib: modify nvpair[@id='virsh_node2-instance_attributes-ipaddr']
  945. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-fenced [1609] (cib_devices_update) info: Updating devices to version 0.13.0
  946. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 3 aborted by virsh_node2-instance_attributes-ipaddr doing modify ipaddr=192.168.122.1: Configuration change | cib=0.13.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='virsh_node2']/instance_attributes[@id='virsh_node2-instance_attributes']/nvpair[@id='virsh_node2-instance_attributes-ipaddr'] complete=false
  947. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-fenced [1609] (stonith_device_register) notice: Added 'virsh_node2' to the device list (2 active devices)
  948. Jul 14 21:41:58 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.13.0)
  949. Jul 14 21:41:59 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-14.raw
  950. Jul 14 21:41:59 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Wrote version 0.13.0 of the CIB to disk (digest: 6876636a76b471e9db232dbf63bd5909)
  951. Jul 14 21:41:59 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.l6FZuD (digest: /var/lib/pacemaker/cib/cib.RR0dvX)
  952. Jul 14 21:42:03 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_ping) info: Reporting our current digest to el8-a01n02: a781c069a7af021eb6ff95d7fc802f5d for 0.13.0 (0x55e7a1d0a4a0 0)
  953. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.0 2
  954. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.1 (null)
  955. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=1
  956. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:1;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=27, @rc-code=1, @op-status=0, @exec-time=11422
  957. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']: <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="0:1;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a01n01" call
  958. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/34, version=0.13.1)
  959. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 3 aborted by operation virsh_node2_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 cib=0.13.1 source=process_graph_event:453 complete=false
  960. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (update_failcount) info: Updating failcount for virsh_node2 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762928)
  961. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) notice: Transition 3 action 7 (virsh_node2_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=27
  962. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n02
  963. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 10 with 2 changes for fail-count-virsh_node2#start_0 (id n/a, set n/a)
  964. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n01]: (unset) -> 1594762928 | from el8-a01n02
  965. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 11 with 2 changes for last-failure-virsh_node2#start_0 (id n/a, set n/a)
  966. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (run_graph) notice: Transition 3 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-3.bz2): Complete
  967. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  968. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/10)
  969. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/11)
  970. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.1 2
  971. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.2 (null)
  972. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=2
  973. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
  974. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/10, version=0.13.2)
  975. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 10 result for fail-count-virsh_node2#start_0: OK | rc=0
  976. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n01]=INFINITY
  977. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n02]=INFINITY
  978. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 3 aborted by status-1-fail-count-virsh_node2.start_0 doing create fail-count-virsh_node2#start_0=INFINITY: Transient attribute change | cib=0.13.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  979. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.2 2
  980. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.3 (null)
  981. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=3
  982. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762928"/>
  983. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/11, version=0.13.3)
  984. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 3 aborted by status-1-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762928: Transient attribute change | cib=0.13.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  985. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 11 result for last-failure-virsh_node2#start_0: OK | rc=0
  986. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n01]=1594762928
  987. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n02]=1594762917
  988. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  989. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  990. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  991. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  992. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node2_start_0 on el8-a01n01 changed: was 4a4e60cff47c9941de05fededd1d6f91 vs. now 229f68a740b52f32c898827417d3a5d2 (reload:3.3.0) 0:1;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  993. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node2 on el8-a01n01 because resource parameters have changed | virsh_node2_clear_failcount_0
  994. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:41:57 2020 | rc=1 id=virsh_node2_last_failure_0
  995. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:41:57 2020 | rc=1 id=virsh_node2_last_0
  996. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node2_start_0 on el8-a01n02 changed: was 4a4e60cff47c9941de05fededd1d6f91 vs. now 229f68a740b52f32c898827417d3a5d2 (reload:3.3.0) 0:1;11:0:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  997. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node2 on el8-a01n02 because resource parameters have changed | virsh_node2_clear_failcount_0
  998. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:41:45 2020 | rc=1 id=virsh_node2_last_failure_0
  999. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1000. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1001. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1002. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1003. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  1004. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n01
  1005. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1006. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1007. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  1008. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1009. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (rsc_action_digest_cmp) info: Parameters to virsh_node2_start_0 on el8-a01n01 changed: was 4a4e60cff47c9941de05fededd1d6f91 vs. now 229f68a740b52f32c898827417d3a5d2 (reload:3.3.0) 0:1;7:3:0:11323fbf-740e-4c7f-8ee5-4d98173fee16
  1010. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (check_action_definition) info: params:reload <parameters login="root" port="el8-a01n02" pcmk_host_list="el8-a01n02" password_script="/root/stonith.pw" ipaddr="192.168.122.1"/>
  1011. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1012. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pe__clear_failcount) notice: Clearing failure of virsh_node2 on el8-a01n01 because action definition changed | virsh_node2_clear_failcount_0
  1013. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  1014. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave virsh_node1 (Started el8-a01n01)
  1015. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Recover virsh_node2 ( el8-a01n01 -> el8-a01n02 )
  1016. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1017. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  1018. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1019. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  1020. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-4.bz2
  1021. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1022. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 4 (ref=pe_calc-dc-1594762928-38) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2
  1023. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_crm_command) info: Executing crm-event (6) locally without waiting: clear_failcount on el8-a01n02
  1024. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n01 | action 3
  1025. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_crm_command) info: Executing crm-event (2) without waiting: clear_failcount on el8-a01n01
  1026. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for virsh_node2 on cluster node el8-a01n02
  1027. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='el8-a01n02']/lrm/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0'] to all (origin=local/crmd/57)
  1028. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n02]: INFINITY -> (unset) | from el8-a01n02
  1029. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 12 with 2 changes for fail-count-virsh_node2#start_0 (id n/a, set n/a)
  1030. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n02]: 1594762917 -> (unset) | from el8-a01n02
  1031. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 13 with 2 changes for last-failure-virsh_node2#start_0 (id n/a, set n/a)
  1032. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.3 2
  1033. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.4 (null)
  1034. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=4
  1035. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=3:4:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;3:4:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762928, @last-run=1594762928, @exec-time=0
  1036. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n01]: INFINITY -> (unset) | from el8-a01n01
  1037. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Write out of 'fail-count-virsh_node2#start_0' delayed: update 12 in progress
  1038. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/35, version=0.13.4)
  1039. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n01]: 1594762928 -> (unset) | from el8-a01n01
  1040. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Write out of 'last-failure-virsh_node2#start_0' delayed: update 13 in progress
  1041. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.4 2
  1042. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.5 (null)
  1043. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0']
  1044. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=5
  1045. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='el8-a01n02']/lrm/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0']: OK (rc=0, origin=el8-a01n02/crmd/57, version=0.13.5)
  1046. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/12)
  1047. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/13)
  1048. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 4 aborted by deletion of lrm_rsc_op[@id='virsh_node2_last_failure_0']: Resource operation removal | cib=0.13.5 source=process_op_deletion:374 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0'] complete=false
  1049. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.5 2
  1050. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.6 (null)
  1051. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-virsh_node2.start_0']
  1052. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=6
  1053. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/12, version=0.13.6)
  1054. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 12 result for fail-count-virsh_node2#start_0: OK | rc=0
  1055. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n01]=INFINITY
  1056. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n02]=(null)
  1057. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 14 with 2 changes for fail-count-virsh_node2#start_0 (id n/a, set n/a)
  1058. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.6 2
  1059. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.7 (null)
  1060. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-virsh_node2.start_0']
  1061. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=7
  1062. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/13, version=0.13.7)
  1063. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 13 result for last-failure-virsh_node2#start_0: OK | rc=0
  1064. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n01]=1594762928
  1065. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n02]=(null)
  1066. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 15 with 2 changes for last-failure-virsh_node2#start_0 (id n/a, set n/a)
  1067. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.7 2
  1068. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.8 (null)
  1069. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0']
  1070. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=8
  1071. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='el8-a01n01']/lrm/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0']: OK (rc=0, origin=el8-a01n01/crmd/36, version=0.13.8)
  1072. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.8 2
  1073. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.9 (null)
  1074. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=9
  1075. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;3:4:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=29, @rc-code=0, @op-status=0, @exec-time=1
  1076. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/37, version=0.13.9)
  1077. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 4 aborted by deletion of nvpair[@id='status-2-fail-count-virsh_node2.start_0']: Transient attribute change | cib=0.13.6 source=abort_unless_down:343 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-virsh_node2.start_0'] complete=false
  1078. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 4 aborted by deletion of nvpair[@id='status-2-last-failure-virsh_node2.start_0']: Transient attribute change | cib=0.13.7 source=abort_unless_down:343 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-virsh_node2.start_0'] complete=false
  1079. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/14)
  1080. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/15)
  1081. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.9 2
  1082. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.10 (null)
  1083. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-virsh_node2.start_0']
  1084. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=10
  1085. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/14, version=0.13.10)
  1086. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 14 result for fail-count-virsh_node2#start_0: OK | rc=0
  1087. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n01]=(null)
  1088. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * fail-count-virsh_node2#start_0[el8-a01n02]=(null)
  1089. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 4 aborted by deletion of lrm_rsc_op[@id='virsh_node2_last_failure_0']: Resource operation removal | cib=0.13.8 source=process_op_deletion:374 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0'] complete=false
  1090. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 4 action 3 (virsh_node2_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=29
  1091. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 4 aborted by deletion of nvpair[@id='status-1-fail-count-virsh_node2.start_0']: Transient attribute change | cib=0.13.10 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-virsh_node2.start_0'] complete=false
  1092. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (run_graph) notice: Transition 4 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Stopped
  1093. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1094. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.10 2
  1095. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.11 (null)
  1096. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-virsh_node2.start_0']
  1097. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=11
  1098. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/15, version=0.13.11)
  1099. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 15 result for last-failure-virsh_node2#start_0: OK | rc=0
  1100. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n01]=(null)
  1101. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * last-failure-virsh_node2#start_0[el8-a01n02]=(null)
  1102. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 4 aborted by deletion of nvpair[@id='status-1-last-failure-virsh_node2.start_0']: Transient attribute change | cib=0.13.11 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-virsh_node2.start_0'] complete=true
  1103. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  1104. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  1105. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n02 is active
  1106. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is online
  1107. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1108. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1109. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1110. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1111. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  1112. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1113. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1114. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1115. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  1116. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1117. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  1118. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave virsh_node1 (Started el8-a01n01)
  1119. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Start virsh_node2 ( el8-a01n02 )
  1120. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1121. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  1122. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1123. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  1124. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-5.bz2
  1125. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1126. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 5 (ref=pe_calc-dc-1594762928-42) derived from /var/lib/pacemaker/pengine/pe-input-5.bz2
  1127. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n02 | action 8
  1128. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=8:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_start_0
  1129. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/60)
  1130. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=26, reply=1, notify=0
  1131. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:virsh_node2 action:start call_id:26
  1132. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.11 2
  1133. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.12 (null)
  1134. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=12
  1135. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=8:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;8:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762928, @last-run=1594762928, @exec-time=0,
  1136. Jul 14 21:42:08 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/60, version=0.13.12)
  1137. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) info: finished - rsc:virsh_node2 action:start call_id:26 exit-code:0 exec-time:696ms queue-time:0ms
  1138. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of start operation for virsh_node2 on el8-a01n02: 0 (ok) | call=26 key=virsh_node2_start_0 confirmed=true cib-update=61
  1139. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61)
  1140. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.12 2
  1141. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.13 (null)
  1142. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=13
  1143. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;8:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=26, @rc-code=0, @op-status=0, @exec-time=696
  1144. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/61, version=0.13.13)
  1145. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 5 action 8 (virsh_node2_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=26
  1146. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_60000 locally on el8-a01n02 | action 9
  1147. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=9:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_monitor_60000
  1148. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=27, reply=1, notify=0
  1149. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:27
  1150. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/62)
  1151. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.13 2
  1152. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.14 (null)
  1153. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=14
  1154. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']: <lrm_rsc_op id="virsh_node2_monitor_60000" operation_key="virsh_node2_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="9:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" transition-magic="-1:193;9:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16" exit-reason="" on_node="el8-a0
  1155. Jul 14 21:42:09 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/62, version=0.13.14)
  1156. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:623ms queue-time:0ms
  1157. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of monitor operation for virsh_node2 on el8-a01n02: 0 (ok) | call=27 key=virsh_node2_monitor_60000 confirmed=false cib-update=63
  1158. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/63)
  1159. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.14 2
  1160. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.15 (null)
  1161. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=15
  1162. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_monitor_60000']: @transition-magic=0:0;9:5:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=27, @rc-code=0, @op-status=0, @exec-time=623
  1163. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/63, version=0.13.15)
  1164. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 5 action 9 (virsh_node2_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=27
  1165. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-controld [1613] (run_graph) notice: Transition 5 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-5.bz2): Complete
  1166. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
  1167. Jul 14 21:42:10 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
  1168. Jul 14 21:42:15 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_ping) info: Reporting our current digest to el8-a01n02: 2bcc92dbb408e1ac1b2596aea324d92c for 0.13.15 (0x55e7a1eab050 0)
  1169. Jul 14 21:42:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1170. Jul 14 21:42:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1171. Jul 14 21:42:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1172. Jul 14 21:42:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1173. Jul 14 21:42:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1174. Jul 14 21:42:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1175. Jul 14 21:42:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1176. Jul 14 21:42:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1177. Jul 14 21:43:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:27
  1178. Jul 14 21:43:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:659ms queue-time:0ms
  1179. Jul 14 21:43:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1180. Jul 14 21:43:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1181. Jul 14 21:43:47 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1182. Jul 14 21:43:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1183. Jul 14 21:43:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1184. Jul 14 21:43:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1185. Jul 14 21:43:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1186. Jul 14 21:43:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1187. Jul 14 21:44:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:27
  1188. Jul 14 21:44:11 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:1020ms queue-time:0ms
  1189. Jul 14 21:44:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1190. Jul 14 21:44:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1191. Jul 14 21:44:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1192. Jul 14 21:44:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1193. Jul 14 21:44:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1194. Jul 14 21:44:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1195. Jul 14 21:44:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1196. Jul 14 21:44:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1197. Jul 14 21:45:11 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:27
  1198. Jul 14 21:45:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:666ms queue-time:0ms
  1199. Jul 14 21:45:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1200. Jul 14 21:45:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1201. Jul 14 21:45:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1202. Jul 14 21:45:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1203. Jul 14 21:45:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1204. Jul 14 21:45:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1205. Jul 14 21:45:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1206. Jul 14 21:45:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1207. Jul 14 21:46:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:27
  1208. Jul 14 21:46:13 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:614ms queue-time:0ms
  1209. Jul 14 21:46:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1210. Jul 14 21:46:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1211. Jul 14 21:46:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1212. Jul 14 21:46:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1213. Jul 14 21:46:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1214. Jul 14 21:46:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1215. Jul 14 21:46:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1216. Jul 14 21:46:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1217. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemakerd [1607] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  1218. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemakerd [1607] (pcmk_shutdown_worker) notice: Shutting down Pacemaker
  1219. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemakerd [1607] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 1613
  1220. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  1221. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms
  1222. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_log) info: Input I_SHUTDOWN received in state S_IDLE from crm_shutdown
  1223. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown
  1224. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n02)
  1225. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was member)
  1226. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (handle_shutdown_request) info: Creating shutdown request for el8-a01n02 (state=S_POLICY_ENGINE)
  1227. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594763213 | from el8-a01n02
  1228. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 16 with 1 change for shutdown (id n/a, set n/a)
  1229. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/16)
  1230. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.15 2
  1231. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.16 (null)
  1232. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=16
  1233. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-shutdown" name="shutdown" value="1594763213"/>
  1234. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/16, version=0.13.16)
  1235. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) info: Transition 5 aborted by status-2-shutdown doing create shutdown=1594763213: Transient attribute change | cib=0.13.16 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1236. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 16 result for shutdown: OK | rc=0
  1237. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * shutdown[el8-a01n02]=1594763213
  1238. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status_fencing) info: Node el8-a01n01 is active
  1239. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n01 is online
  1240. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (determine_online_status) info: Node el8-a01n02 is shutting down
  1241. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1242. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1243. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 1 is already processed
  1244. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (unpack_node_loop) info: Node 2 is already processed
  1245. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  1246. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (common_print) info: virsh_node2 (stonith:fence_virsh): Started el8-a01n02
  1247. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1248. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1249. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (clone_print) info: Clone Set: drbd-clone [drbd]
  1250. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1251. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1252. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (native_color) info: Resource drbd:1 cannot run anywhere
  1253. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  1254. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n02
  1255. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogNodeActions) notice: * Shutdown el8-a01n02
  1256. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave virsh_node1 (Started el8-a01n01)
  1257. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Move virsh_node2 ( el8-a01n02 -> el8-a01n01 )
  1258. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1259. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Stop hypervisor:1 ( el8-a01n02 ) due to node availability
  1260. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1261. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1262. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-schedulerd[1612] (pcmk__log_transition_summary) notice: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-6.bz2
  1263. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1264. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_te_invoke) info: Processing graph 6 (ref=pe_calc-dc-1594763213-46) derived from /var/lib/pacemaker/pengine/pe-input-6.bz2
  1265. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n02 | action 9
  1266. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:27 exit-code:0 exec-time:0ms queue-time:0ms
  1267. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  1268. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=9:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=virsh_node2_stop_0
  1269. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=29, reply=1, notify=0
  1270. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:29
  1271. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) info: Result of monitor operation for virsh_node2 on el8-a01n02: Cancelled | call=27 key=virsh_node2_monitor_60000 confirmed=true
  1272. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n02 | action 14
  1273. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:29 exit-code:0 exec-time:1ms queue-time:0ms
  1274. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000
  1275. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/65)
  1276. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:22 exit-code:0 exec-time:0ms queue-time:0ms
  1277. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  1278. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=14:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=hypervisor_stop_0
  1279. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/66)
  1280. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=31, reply=1, notify=0
  1281. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:hypervisor action:stop call_id:31
  1282. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit libvirtd named 'hypervisor'
  1283. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.16 2
  1284. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.17 (null)
  1285. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=17
  1286. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=9:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;9:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763213, @last-run=1594763213, @exec-time=0
  1287. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/65, version=0.13.17)
  1288. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec_with_unit) debug: Calling StopUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
  1289. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.17 2
  1290. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.18 (null)
  1291. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=18
  1292. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=14:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;14:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763213, @last-run=1594763213, @exec-time=0
  1293. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/66, version=0.13.18)
  1294. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 locally on el8-a01n02 | action 21
  1295. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (cancel_recurring_action) info: Cancelling systemd operation drbd_status_60000
  1296. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:drbd action:monitor call_id:24 exit-code:0 exec-time:0ms queue-time:0ms
  1297. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=0, reply=1, notify=0
  1298. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (do_lrm_rsc_op) info: Performing key=21:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16 op=drbd_stop_0
  1299. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/67)
  1300. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 74d60a7f-ef31-4534-8c88-bc870fd67528: rc=33, reply=1, notify=0
  1301. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) info: executing - rsc:drbd action:stop call_id:33
  1302. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit drbd named 'drbd'
  1303. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.18 2
  1304. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.19 (null)
  1305. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=19
  1306. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=21:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @transition-magic=-1:193;21:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594763213, @last-run=1594763213, @exec-time=0
  1307. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/67, version=0.13.19)
  1308. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n02: 0 (ok) | call=29 key=virsh_node2_stop_0 confirmed=true cib-update=68
  1309. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n02: Cancelled | call=22 key=hypervisor_monitor_60000 confirmed=true
  1310. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) info: Result of monitor operation for drbd on el8-a01n02: Cancelled | call=24 key=drbd_monitor_60000 confirmed=true
  1311. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/68)
  1312. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member)
  1313. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_TRANSITION_ENGINE)
  1314. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594763213 | from el8-a01n02
  1315. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (write_attribute) info: Sent CIB request 17 with 2 changes for shutdown (id n/a, set n/a)
  1316. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.19 2
  1317. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.20 (null)
  1318. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=20
  1319. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;9:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=29, @rc-code=0, @op-status=0, @exec-time=1
  1320. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 6 action 9 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=29
  1321. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 on el8-a01n01 | action 10
  1322. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/68, version=0.13.20)
  1323. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 6 aborted by lrm_rsc_op.virsh_node2_last_failure_0: Event failed | cib=0.0.0 source=process_graph_event:453 path=/create_request_adv/crm_xml/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_failure_0'] complete=false
  1324. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 6 action 10 (virsh_node2_start_0 on el8-a01n01): Cannot execute now
  1325. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_TRANSITION_ENGINE)
  1326. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/17)
  1327. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.20 2
  1328. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.21 (null)
  1329. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=21
  1330. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-shutdown" name="shutdown" value="1594763213"/>
  1331. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-controld [1613] (abort_transition_graph) notice: Transition 6 aborted by status-1-shutdown doing create shutdown=1594763213: Transient attribute change | cib=0.13.21 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false
  1332. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/17, version=0.13.21)
  1333. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: CIB update 17 result for shutdown: OK | rc=0
  1334. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * shutdown[el8-a01n01]=1594763213
  1335. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-attrd [1611] (attrd_cib_callback) info: * shutdown[el8-a01n02]=1594763213
  1336. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1219
  1337. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: hypervisor stop may still be in progress: re-scheduling (elapsed=203ms, remaining=99797ms, start_delay=2000ms)
  1338. Jul 14 21:46:53 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec_with_unit) debug: Calling StopUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice
  1339. Jul 14 21:46:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/1220
  1340. Jul 14 21:46:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=355ms, remaining=99645ms, start_delay=2000ms)
  1341. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:31
  1342. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1343. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1344. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: hypervisor stop is now complete (elapsed=2208ms, remaining=97792ms): not running (7)
  1345. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:31 exit-code:0 exec-time:2208ms queue-time:0ms
  1346. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/69)
  1347. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n02: 0 (ok) | call=31 key=hypervisor_stop_0 confirmed=true cib-update=69
  1348. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: --- 0.13.21 2
  1349. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: Diff: +++ 0.13.22 (null)
  1350. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib: @num_updates=22
  1351. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;14:6:0:11323fbf-740e-4c7f-8ee5-4d98173fee16, @call-id=31, @rc-code=0, @op-status=0, @exec-time=2208
  1352. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-controld [1613] (process_graph_event) info: Transition 6 action 14 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=31
  1353. Jul 14 21:46:55 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/69, version=0.13.22)
  1354. Jul 14 21:46:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1355. Jul 14 21:46:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1356. Jul 14 21:46:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1357. Jul 14 21:46:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=2367ms, remaining=97633ms, start_delay=2000ms)
  1358. Jul 14 21:46:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1359. Jul 14 21:46:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1360. Jul 14 21:46:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1361. Jul 14 21:46:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=4370ms, remaining=95630ms, start_delay=2000ms)
  1362. Jul 14 21:47:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1363. Jul 14 21:47:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1364. Jul 14 21:47:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1365. Jul 14 21:47:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=6374ms, remaining=93626ms, start_delay=2000ms)
  1366. Jul 14 21:47:00 el8-a01n02.digimer.ca pacemaker-based [1608] (cib_process_ping) info: Reporting our current digest to el8-a01n02: 416cfb2580c7286c9b7cd6f5bff3da81 for 0.13.22 (0x55e7a1eab050 0)
  1367. Jul 14 21:47:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1368. Jul 14 21:47:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1369. Jul 14 21:47:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1370. Jul 14 21:47:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=8378ms, remaining=91622ms, start_delay=2000ms)
  1371. Jul 14 21:47:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1372. Jul 14 21:47:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1373. Jul 14 21:47:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1374. Jul 14 21:47:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=10380ms, remaining=89620ms, start_delay=2000ms)
  1375. Jul 14 21:47:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1376. Jul 14 21:47:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1377. Jul 14 21:47:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1378. Jul 14 21:47:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=12384ms, remaining=87616ms, start_delay=2000ms)
  1379. Jul 14 21:47:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1380. Jul 14 21:47:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1381. Jul 14 21:47:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1382. Jul 14 21:47:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=14387ms, remaining=85613ms, start_delay=2000ms)
  1383. Jul 14 21:47:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1384. Jul 14 21:47:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1385. Jul 14 21:47:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1386. Jul 14 21:47:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=16390ms, remaining=83610ms, start_delay=2000ms)
  1387. Jul 14 21:47:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1388. Jul 14 21:47:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1389. Jul 14 21:47:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1390. Jul 14 21:47:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=18403ms, remaining=81597ms, start_delay=2000ms)
  1391. Jul 14 21:47:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1392. Jul 14 21:47:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1393. Jul 14 21:47:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1394. Jul 14 21:47:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=20406ms, remaining=79594ms, start_delay=2000ms)
  1395. Jul 14 21:47:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1396. Jul 14 21:47:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1397. Jul 14 21:47:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1398. Jul 14 21:47:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=22409ms, remaining=77591ms, start_delay=2000ms)
  1399. Jul 14 21:47:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1400. Jul 14 21:47:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1401. Jul 14 21:47:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1402. Jul 14 21:47:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=24413ms, remaining=75587ms, start_delay=2000ms)
  1403. Jul 14 21:47:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1404. Jul 14 21:47:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1405. Jul 14 21:47:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1406. Jul 14 21:47:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=26416ms, remaining=73584ms, start_delay=2000ms)
  1407. Jul 14 21:47:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1408. Jul 14 21:47:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1409. Jul 14 21:47:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1410. Jul 14 21:47:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=28420ms, remaining=71580ms, start_delay=2000ms)
  1411. Jul 14 21:47:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1412. Jul 14 21:47:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1413. Jul 14 21:47:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1414. Jul 14 21:47:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=30423ms, remaining=69577ms, start_delay=2000ms)
  1415. Jul 14 21:47:26 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1416. Jul 14 21:47:26 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1417. Jul 14 21:47:26 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1418. Jul 14 21:47:26 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=32427ms, remaining=67573ms, start_delay=2000ms)
  1419. Jul 14 21:47:28 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1420. Jul 14 21:47:28 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1421. Jul 14 21:47:28 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1422. Jul 14 21:47:28 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=34430ms, remaining=65570ms, start_delay=2000ms)
  1423. Jul 14 21:47:30 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1424. Jul 14 21:47:30 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1425. Jul 14 21:47:30 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1426. Jul 14 21:47:30 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=36434ms, remaining=63566ms, start_delay=2000ms)
  1427. Jul 14 21:47:32 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1428. Jul 14 21:47:32 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1429. Jul 14 21:47:32 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1430. Jul 14 21:47:32 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=38441ms, remaining=61559ms, start_delay=2000ms)
  1431. Jul 14 21:47:34 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1432. Jul 14 21:47:34 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1433. Jul 14 21:47:34 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1434. Jul 14 21:47:34 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=40444ms, remaining=59556ms, start_delay=2000ms)
  1435. Jul 14 21:47:36 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1436. Jul 14 21:47:36 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1437. Jul 14 21:47:36 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1438. Jul 14 21:47:36 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=42447ms, remaining=57553ms, start_delay=2000ms)
  1439. Jul 14 21:47:38 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1440. Jul 14 21:47:38 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1441. Jul 14 21:47:38 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1442. Jul 14 21:47:38 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=44451ms, remaining=55549ms, start_delay=2000ms)
  1443. Jul 14 21:47:40 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1444. Jul 14 21:47:40 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1445. Jul 14 21:47:40 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1446. Jul 14 21:47:40 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=46455ms, remaining=53545ms, start_delay=2000ms)
  1447. Jul 14 21:47:42 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1448. Jul 14 21:47:42 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1449. Jul 14 21:47:42 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1450. Jul 14 21:47:42 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=48458ms, remaining=51542ms, start_delay=2000ms)
  1451. Jul 14 21:47:44 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1452. Jul 14 21:47:44 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1453. Jul 14 21:47:44 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1454. Jul 14 21:47:44 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=50461ms, remaining=49539ms, start_delay=2000ms)
  1455. Jul 14 21:47:46 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1456. Jul 14 21:47:46 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1457. Jul 14 21:47:46 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1458. Jul 14 21:47:46 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=52464ms, remaining=47536ms, start_delay=2000ms)
  1459. Jul 14 21:47:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1460. Jul 14 21:47:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1461. Jul 14 21:47:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1462. Jul 14 21:47:48 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=54467ms, remaining=45533ms, start_delay=2000ms)
  1463. Jul 14 21:47:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1464. Jul 14 21:47:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1465. Jul 14 21:47:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1466. Jul 14 21:47:50 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=56470ms, remaining=43530ms, start_delay=2000ms)
  1467. Jul 14 21:47:52 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1468. Jul 14 21:47:52 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1469. Jul 14 21:47:52 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1470. Jul 14 21:47:52 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=58474ms, remaining=41526ms, start_delay=2000ms)
  1471. Jul 14 21:47:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1472. Jul 14 21:47:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1473. Jul 14 21:47:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1474. Jul 14 21:47:54 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=60477ms, remaining=39523ms, start_delay=2000ms)
  1475. Jul 14 21:47:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1476. Jul 14 21:47:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1477. Jul 14 21:47:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1478. Jul 14 21:47:56 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=62480ms, remaining=37520ms, start_delay=2000ms)
  1479. Jul 14 21:47:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1480. Jul 14 21:47:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1481. Jul 14 21:47:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1482. Jul 14 21:47:58 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=64483ms, remaining=35517ms, start_delay=2000ms)
  1483. Jul 14 21:48:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1484. Jul 14 21:48:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1485. Jul 14 21:48:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1486. Jul 14 21:48:00 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=66486ms, remaining=33514ms, start_delay=2000ms)
  1487. Jul 14 21:48:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1488. Jul 14 21:48:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1489. Jul 14 21:48:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1490. Jul 14 21:48:02 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=68489ms, remaining=31511ms, start_delay=2000ms)
  1491. Jul 14 21:48:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1492. Jul 14 21:48:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1493. Jul 14 21:48:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1494. Jul 14 21:48:04 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=70491ms, remaining=29509ms, start_delay=2000ms)
  1495. Jul 14 21:48:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1496. Jul 14 21:48:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1497. Jul 14 21:48:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1498. Jul 14 21:48:06 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=72499ms, remaining=27501ms, start_delay=2000ms)
  1499. Jul 14 21:48:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1500. Jul 14 21:48:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1501. Jul 14 21:48:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1502. Jul 14 21:48:08 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=74506ms, remaining=25494ms, start_delay=2000ms)
  1503. Jul 14 21:48:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1504. Jul 14 21:48:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1505. Jul 14 21:48:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1506. Jul 14 21:48:10 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=76510ms, remaining=23490ms, start_delay=2000ms)
  1507. Jul 14 21:48:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1508. Jul 14 21:48:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1509. Jul 14 21:48:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1510. Jul 14 21:48:12 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=78513ms, remaining=21487ms, start_delay=2000ms)
  1511. Jul 14 21:48:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1512. Jul 14 21:48:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1513. Jul 14 21:48:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1514. Jul 14 21:48:14 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=80517ms, remaining=19483ms, start_delay=2000ms)
  1515. Jul 14 21:48:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1516. Jul 14 21:48:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1517. Jul 14 21:48:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1518. Jul 14 21:48:16 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=82521ms, remaining=17479ms, start_delay=2000ms)
  1519. Jul 14 21:48:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1520. Jul 14 21:48:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1521. Jul 14 21:48:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1522. Jul 14 21:48:18 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=84524ms, remaining=15476ms, start_delay=2000ms)
  1523. Jul 14 21:48:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1524. Jul 14 21:48:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1525. Jul 14 21:48:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1526. Jul 14 21:48:20 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=86531ms, remaining=13469ms, start_delay=2000ms)
  1527. Jul 14 21:48:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1528. Jul 14 21:48:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1529. Jul 14 21:48:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1530. Jul 14 21:48:22 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=88534ms, remaining=11466ms, start_delay=2000ms)
  1531. Jul 14 21:48:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (log_execute) debug: executing - rsc:drbd action:monitor call_id:33
  1532. Jul 14 21:48:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1533. Jul 14 21:48:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1534. Jul 14 21:48:24 el8-a01n02.digimer.ca pacemaker-execd [1610] (action_complete) debug: drbd stop may still be in progress: re-scheduling (elapsed=90537ms, remaining=9463ms, start_delay=2000ms)
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!