API tools faq
paste
digimer

Untitled

digimer
Jul 14th, 2020
161
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 340.82 KB | None | 0 0
raw download clone embed print report
  1. [root@el8-a01n01 ~]# cat /var/log/pacemaker/pacemaker.log
  2. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_log_init) info: Cannot change active directory to /var/lib/pacemaker/cores: No such file or directory (2)
  3. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (get_cluster_type) info: Detected an active 'corosync' cluster
  4. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (mcp_read_config) info: Reading configure for stack: corosync
  5. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (main) notice: Starting Pacemaker 2.0.3-5.el8_2.1 | build=4b1f869f0f features: generated-manpages agent-manpages ncurses libqb-logging libqb-ipc systemd nagios corosync-native atomic-attrd acls
  6. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (main) info: Maximum core file size is: 18446744073709551615
  7. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (qb_ipcs_us_publish) info: server name: pacemakerd
  8. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to lrmd IPC: Connection refused
  9. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to cib_ro IPC: Connection refused
  10. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to crmd IPC: Connection refused
  11. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection refused
  12. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to pengine IPC: Connection refused
  13. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk__ipc_is_authentic_process_active) info: Could not connect to stonith-ng IPC: Connection refused
  14. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Created entry 724e6ee6-fc51-496f-b6d2-093b6b017a7b/0x55dbca2ebf90 for node el8-a01n01/1 (1 total)
  15. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  16. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Node 1 has uuid 1
  17. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
  18. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (cluster_connect_quorum) warning: Quorum lost
  19. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Created entry c7daba3d-a73b-4929-9d26-7ec6ef2a18a7/0x55dbca4ed8b0 for node el8-a01n02/2 (2 total)
  20. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  21. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_get_peer) info: Node 2 has uuid 2
  22. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Using uid=189 and group=189 for process pacemaker-based
  23. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375409 for process pacemaker-based
  24. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375410 for process pacemaker-fenced
  25. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375411 for process pacemaker-execd
  26. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Using uid=189 and group=189 for process pacemaker-attrd
  27. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375412 for process pacemaker-attrd
  28. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Using uid=189 and group=189 for process pacemaker-schedulerd
  29. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375413 for process pacemaker-schedulerd
  30. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Using uid=189 and group=189 for process pacemaker-controld
  31. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (start_child) info: Forked child 375414 for process pacemaker-controld
  32. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (main) notice: Pacemaker daemon successfully started and accepting connections
  33. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_quorum_notification) info: Quorum still lost | membership=140 members=1
  34. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
  35. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (crm_reap_unseen_nodes) info: State of node el8-a01n02[2] is still unknown
  36. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_cpg_membership) info: Group pacemakerd event 0: node 1 pid 375401 joined via cpg_join
  37. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_cpg_membership) info: Group pacemakerd event 0: el8-a01n01 (node 1 pid 375401) is member
  38. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  39. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-attrd [375412] (main) notice: Starting Pacemaker node attribute manager
  40. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  41. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (main) notice: Starting Pacemaker CIB manager
  42. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (get_cluster_type) info: Verifying cluster type: 'corosync'
  43. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (get_cluster_type) info: Assuming an active 'corosync' cluster
  44. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk__daemon_can_write) notice: /var/lib/pacemaker/cib/cib.xml not found: No such file or directory
  45. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk__daemon_can_write) notice: /var/lib/pacemaker/cib/cib.xml.sig not found: No such file or directory
  46. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (retrieveCib) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
  47. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_read_and_verify) warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)
  48. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (readCibXmlFile) warning: Primary configuration corrupt or unusable, trying backups in /var/lib/pacemaker/cib
  49. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (readCibXmlFile) warning: Continuing with an empty configuration.
  50. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd [375411] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  51. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd [375411] (main) notice: Starting Pacemaker local executor
  52. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_ipcs_us_publish) info: server name: lrmd
  53. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-execd [375411] (main) notice: Pacemaker local executor successfully started and accepting connections
  54. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  55. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  56. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld [375414] (main) notice: Starting Pacemaker controller
  57. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_log) info: Input I_STARTUP received in state S_STARTING from crmd_init
  58. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  59. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (main) notice: Starting Pacemaker fencer
  60. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (get_cluster_type) info: Verifying cluster type: 'corosync'
  61. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (get_cluster_type) info: Assuming an active 'corosync' cluster
  62. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  63. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Created entry 64b0555c-0df6-469b-b304-b2214c0c1b76/0x555fc5d7ed60 for node el8-a01n01/1 (1 total)
  64. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  65. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores
  66. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (main) notice: Starting Pacemaker scheduler
  67. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (qb_ipcs_us_publish) info: server name: pengine
  68. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (main) notice: Pacemaker scheduler successfully started and accepting connections
  69. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Created entry 1b70802f-7e5a-434f-8a6a-5c3b48769ac1/0x555a63b7b7a0 for node el8-a01n01/1 (1 total)
  70. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  71. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Node 1 has uuid 1
  72. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
  73. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  74. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (init_cs_connection_once) info: Connection to 'corosync': established
  75. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (qb_ipcs_us_publish) info: server name: cib_ro
  76. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (qb_ipcs_us_publish) info: server name: cib_rw
  77. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (qb_ipcs_us_publish) info: server name: cib_shm
  78. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (main) notice: Pacemaker CIB manager successfully started and accepting connections
  79. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk_cpg_membership) info: Group cib event 0: node 1 pid 375409 joined via cpg_join
  80. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk_cpg_membership) info: Group cib event 0: el8-a01n01 (node 1 pid 375409) is member
  81. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Node 1 has uuid 1
  82. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
  83. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  84. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (init_cs_connection_once) info: Connection to 'corosync': established
  85. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_read_and_verify) warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2)
  86. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw
  87. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (setup_cib) info: Watching for stonith topology changes
  88. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (qb_ipcs_us_publish) info: server name: stonith-ng
  89. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (main) notice: Pacemaker fencer successfully started and accepting connections
  90. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (pcmk_cpg_membership) info: Group stonith-ng event 0: node 1 pid 375410 joined via cpg_join
  91. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (pcmk_cpg_membership) info: Group stonith-ng event 0: el8-a01n01 (node 1 pid 375410) is member
  92. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (init_cib_cache_cb) info: Updating device list from the cib: init
  93. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.0.0
  94. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-fenced [375410] (unpack_nodes) info: Creating a fake local node
  95. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.0.0 of the CIB to disk (digest: 2469c254d4b2872e864ff6b9c730ac0c)
  96. Jul 14 21:28:46 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.azN5iO (digest: /var/lib/pacemaker/cib/cib.b8uEay)
  97. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk_cpg_membership) info: Group cib event 1: node 2 pid 4565 joined via cluster join
  98. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 2 pid 4566 joined via cluster join
  99. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n01 (node 1 pid 375410) is member
  100. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_cpg_membership) info: Group pacemakerd event 1: node 2 pid 4564 joined via cluster join
  101. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n01 (node 1 pid 375401) is member
  102. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_cpg_membership) info: Group pacemakerd event 1: el8-a01n02 (node 2 pid 4564) is member
  103. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
  104. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n01 (node 1 pid 375409) is member
  105. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining, last=7): Try again (6)
  106. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_quorum_notification) notice: Quorum acquired | membership=144 members=2
  107. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemakerd [375401] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
  108. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Created entry 985cf59e-a073-4bb7-8a71-1afd8108238b/0x555a63b823f0 for node el8-a01n02/2 (2 total)
  109. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  110. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_get_peer) info: Node 2 has uuid 2
  111. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (pcmk_cpg_membership) info: Group cib event 1: el8-a01n02 (node 2 pid 4565) is member
  112. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
  113. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  114. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Created entry a122cdea-22b5-4c4d-89a1-fc060ffad198/0x555fc5d7c270 for node el8-a01n02/2 (2 total)
  115. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  116. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_get_peer) info: Node 2 has uuid 2
  117. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (pcmk_cpg_membership) info: Group stonith-ng event 1: el8-a01n02 (node 2 pid 4566) is member
  118. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
  119. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-fenced [375410] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  120. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/transient_attributes: OK (rc=0, origin=el8-a01n02/attrd/2, version=0.0.0)
  121. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (main) info: CIB connection active
  122. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (get_cluster_type) info: Verifying cluster type: 'corosync'
  123. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (get_cluster_type) info: Assuming an active 'corosync' cluster
  124. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  125. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Created entry 521b1756-729a-45ed-a585-3cbafa292175/0x55f3182f8a20 for node el8-a01n01/1 (1 total)
  126. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  127. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Node 1 has uuid 1
  128. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
  129. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
  130. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (init_cs_connection_once) info: Connection to 'corosync': established
  131. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (main) info: Cluster connection active
  132. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_erase_attrs) info: Clearing transient attributes from CIB | xpath=//node_state[@uname='el8-a01n01']/transient_attributes
  133. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (qb_ipcs_us_publish) info: server name: attrd
  134. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (main) notice: Pacemaker node attribute manager successfully started and accepting connections
  135. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (pcmk_cpg_membership) info: Group attrd event 0: node 1 pid 375412 joined via cpg_join
  136. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n01 (node 1 pid 375412) is member
  137. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.0.0 2
  138. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.1.0 (null)
  139. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=1
  140. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/nodes: <node id="1" uname="el8-a01n01"/>
  141. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/nodes: <node id="2" uname="el8-a01n02"/>
  142. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n02/crmd/4, version=0.1.0)
  143. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes to all (origin=local/attrd/2)
  144. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Created entry 2162ef55-efca-47dc-95c3-b15f8ac5669b/0x55f3182fb6d0 for node el8-a01n02/2 (2 total)
  145. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  146. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_get_peer) info: Node 2 has uuid 2
  147. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (pcmk_cpg_membership) info: Group attrd event 0: el8-a01n02 (node 2 pid 4568) is member
  148. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
  149. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
  150. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_check_for_new_writer) notice: Recorded new attribute writer: el8-a01n02 (was unset)
  151. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_message) info: Processing sync-response from el8-a01n02
  152. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2 | from el8-a01n02
  153. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2 | from el8-a01n01
  154. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw
  155. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.1.0 of the CIB to disk (digest: 7aefab66c16403ca8289ab3d2d263d82)
  156. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nXt7xa (digest: /var/lib/pacemaker/cib/cib.M3TlhX)
  157. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/transient_attributes: OK (rc=0, origin=el8-a01n01/attrd/2, version=0.1.0)
  158. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (get_cluster_type) info: Verifying cluster type: 'corosync'
  159. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (get_cluster_type) info: Assuming an active 'corosync' cluster
  160. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_cluster_connect) notice: Connecting to cluster infrastructure: corosync
  161. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Created entry c7a1c8a9-639f-4423-9092-8fc91d5806e6/0x558f70425eb0 for node el8-a01n01/1 (1 total)
  162. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Node 1 is now known as el8-a01n01
  163. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Cluster node el8-a01n01 is now in unknown state
  164. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Node 1 has uuid 1
  165. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_proc) info: cluster_connect_cpg: Node el8-a01n01[1] - corosync-cpg is now online
  166. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Node el8-a01n01 is now a peer | DC=<none> old=0x0000000 new=0x4000000
  167. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds
  168. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (init_cs_connection_once) info: Connection to 'corosync': established
  169. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (cluster_connect_quorum) notice: Quorum acquired
  170. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Created entry 6a74f85b-f24c-49ec-acac-ebc0c3e8ad7d/0x558f70428740 for node el8-a01n02/2 (2 total)
  171. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Node 2 is now known as el8-a01n02
  172. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Cluster node el8-a01n02 is now in unknown state
  173. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_get_peer) info: Node 2 has uuid 2
  174. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_ha_control) info: Connected to the cluster
  175. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (lrmd_ipc_connect) info: Connecting to executor
  176. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (crm_client_new) debug: Connecting 0x556a6a524fb0 for uid=189 gid=189 pid=375414 id=09acd970-a9ad-4616-9dfb-40686968c8db
  177. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (handle_new_connection) debug: IPC credentials authenticated (/dev/shm/qb-375411-375414-9-4xxB6U/qb)
  178. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_ipcs_shm_connect) debug: connecting to client [375414]
  179. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  180. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  181. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/4)
  182. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  183. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/4, version=0.1.0)
  184. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  185. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_control) info: Connection to the executor established
  186. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_started) info: Delaying start, no membership data (0000000000100000)
  187. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_started) info: Delaying start, no membership data (0000000000100000)
  188. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (pcmk_quorum_notification) info: Quorum retained | membership=144 members=2
  189. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_state_iter) notice: Node el8-a01n01 state is now member | nodeid=1 previous=unknown source=pcmk_quorum_notification
  190. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Cluster node el8-a01n01 is now member (was in unknown state)
  191. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_state_iter) notice: Node el8-a01n02 state is now member | nodeid=2 previous=unknown source=pcmk_quorum_notification
  192. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Cluster node el8-a01n02 is now member (was in unknown state)
  193. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (qb_ipcs_us_publish) info: server name: crmd
  194. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_started) notice: Pacemaker controller successfully started and accepting connections
  195. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_log) info: Input I_PENDING received in state S_STARTING from do_started
  196. Jul 14 21:28:47 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started
  197. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (pcmk_cpg_membership) info: Group crmd event 0: node 1 pid 375414 joined via cpg_join
  198. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n01 (node 1 pid 375414) is member
  199. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (pcmk_cpg_membership) info: Group crmd event 0: el8-a01n02 (node 2 pid 4570) is member
  200. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_proc) info: pcmk_cpg_membership: Node el8-a01n02[2] - corosync-cpg is now online
  201. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (peer_update_callback) info: Node el8-a01n02 is now a peer | DC=<none> old=0x0000000 new=0x4000000
  202. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds
  203. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 30 seconds
  204. Jul 14 21:28:48 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_connect_stonith) notice: Fencer successfully connected
  205. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
  206. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.1.0 2
  207. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.2.0 (null)
  208. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=2
  209. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <primitive class="stonith" id="virsh_node1" type="fence_virsh"/>
  210. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <instance_attributes id="virsh_node1-instance_attributes">
  211. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-delay" name="delay" value="15"/>
  212. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-ipaddr" name="ipaddr" value="10.20.255.250"/>
  213. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-login" name="login" value="root"/>
  214. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-password_script" name="password_script" value="/root/stonith.pw"/>
  215. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n01"/>
  216. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node1-instance_attributes-port" name="port" value="el8-a01n01"/>
  217. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </instance_attributes>
  218. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  219. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="virsh_node1-monitor-interval-60" interval="60" name="monitor"/>
  220. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  221. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  222. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.2.0)
  223. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  224. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.2.0
  225. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw
  226. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.2.0 of the CIB to disk (digest: 0d075a040dee76aba3d9bb362ddfe793)
  227. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OutjvV (digest: /var/lib/pacemaker/cib/cib.cbWZZA)
  228. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced [375410] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 376630 exited with rc=0
  229. Jul 14 21:29:06 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_device_register) notice: Added 'virsh_node1' to the device list (1 active devices)
  230. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (election_count_vote) info: election-DC round 1 (owner node ID 2) pass: vote from el8-a01n02 (Uptime)
  231. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_PENDING -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote
  232. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (election_check) info: election-DC won by local node
  233. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb
  234. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb
  235. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_control) info: Registering TE UUID: 0df6240b-f9a0-4c7b-bf60-fd2dab145e11
  236. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (set_graph_functions) info: Setting custom graph functions
  237. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_dc_takeover) info: Taking over DC status for this partition
  238. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_readwrite) info: We are now in R/W mode
  239. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.2.0)
  240. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/8)
  241. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/8, version=0.2.0)
  242. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/10)
  243. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.2.0 2
  244. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.3.0 (null)
  245. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=3
  246. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/crm_config: <cluster_property_set id="cib-bootstrap-options"/>
  247. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
  248. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </cluster_property_set>
  249. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/10, version=0.3.0)
  250. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/12)
  251. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.3.0 2
  252. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.4.0 (null)
  253. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=4
  254. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.0.3-5.el8_2.1-4b1f869f0f"/>
  255. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/12, version=0.4.0)
  256. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/14)
  257. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.4.0 2
  258. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.5.0 (null)
  259. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=5
  260. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
  261. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/14, version=0.5.0)
  262. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (join_make_offer) info: Making join-1 offers based on membership event 144
  263. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (join_make_offer) info: Sending join-1 offer to el8-a01n01
  264. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (join_make_offer) info: Sending join-1 offer to el8-a01n02
  265. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_dc_join_offer_all) info: Waiting on join-1 requests from 2 outstanding nodes
  266. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/16)
  267. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.5.0 2
  268. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.0 (null)
  269. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=6
  270. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="el8-anvil-01"/>
  271. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_dc) info: Set DC to el8-a01n01 (3.3.0)
  272. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_expected) info: update_dc: Node el8-a01n01[1] - expected state is now member (was (null))
  273. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_expected) info: do_dc_join_filter_offer: Node el8-a01n02[2] - expected state is now member (was (null))
  274. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=el8-a01n01/crmd/16, version=0.6.0)
  275. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
  276. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (controld_delete_node_state) info: Deleting resource history for node el8-a01n01 (via CIB call 24) | xpath=//node_state[@uname='el8-a01n01']/lrm
  277. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_replace) info: Digest matched on replace from el8-a01n01: dc9656c0a868cfbd5503dbb3659ae0b0
  278. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_replace) info: Replaced 0.6.0 with 0.6.0 from el8-a01n01
  279. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (controld_delete_node_state) info: Deleting resource history for node el8-a01n02 (via CIB call 26) | xpath=//node_state[@uname='el8-a01n02']/lrm
  280. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=el8-a01n01/crmd/21, version=0.6.0)
  281. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/22)
  282. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/23)
  283. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm to all (origin=local/crmd/24)
  284. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/25)
  285. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm to all (origin=local/crmd/26)
  286. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/22, version=0.6.0)
  287. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/23, version=0.6.0)
  288. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n01']/lrm: OK (rc=0, origin=el8-a01n01/crmd/24, version=0.6.0)
  289. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.0 2
  290. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.1 (null)
  291. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=1
  292. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status: <node_state id="1" uname="el8-a01n01" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
  293. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm id="1">
  294. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_resources/>
  295. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm>
  296. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </node_state>
  297. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/25, version=0.6.1)
  298. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
  299. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true
  300. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='el8-a01n02']/lrm: OK (rc=0, origin=el8-a01n01/crmd/26, version=0.6.1)
  301. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27)
  302. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/30)
  303. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31)
  304. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.1 2
  305. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.2 (null)
  306. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=2
  307. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status: <node_state id="2" uname="el8-a01n02" in_ccm="true" crmd="online" crm-debug-origin="do_lrm_query_internal"/>
  308. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm id="2">
  309. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_resources/>
  310. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm>
  311. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </node_state>
  312. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/27, version=0.6.2)
  313. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=el8-a01n01/crmd/30, version=0.6.2)
  314. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.2 2
  315. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.3 (null)
  316. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=3
  317. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition, @join=member, @expected=member
  318. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition, @join=member, @expected=member
  319. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/31, version=0.6.3)
  320. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/32)
  321. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.3 2
  322. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.4 (null)
  323. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=4, @have-quorum=1, @dc-uuid=1
  324. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw
  325. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=el8-a01n01/crmd/32, version=0.6.4)
  326. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.3.0 of the CIB to disk (digest: be8b332ecdd2a858160b9164e32749f2)
  327. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3vARTJ (digest: /var/lib/pacemaker/cib/cib.4ahR0w)
  328. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-4.raw
  329. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.6.0 of the CIB to disk (digest: 90d9c1475d4bcd1028ad0c0f53e4d2ee)
  330. Jul 14 21:29:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HAXpWJ (digest: /var/lib/pacemaker/cib/cib.yoEE5w)
  331. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  332. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  333. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  334. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  335. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  336. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  337. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  338. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  339. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  340. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  341. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start virsh_node1 ( el8-a01n01 )
  342. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-0.bz2
  343. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  344. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 0 (ref=pe_calc-dc-1594762149-9) derived from /var/lib/pacemaker/pengine/pe-input-0.bz2
  345. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n02 | action 2
  346. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n01 | action 1
  347. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node1' not in cache
  348. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  349. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node1'
  350. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
  351. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  352. Jul 14 21:29:09 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_monitor_0
  353. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.4 2
  354. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.5 (null)
  355. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=5
  356. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource
  357. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
  358. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-sta
  359. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  360. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/12, version=0.6.5)
  361. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.5 2
  362. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.6 (null)
  363. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=6
  364. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;2:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=5, @rc-code=7, @op-status=0, @last-rc-change=1594762150, @last-run=1594762150, @exec-time=12
  365. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/13, version=0.6.6)
  366. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (action_synced_wait) info: Managed fence_virsh_metadata_1 process 376820 exited with rc=0
  367. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=5, reply=1, notify=0
  368. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:virsh_node1 action:monitor call_id:5
  369. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (stonith_api_signon) debug: Attempting fencer connection by pacemaker-execd with mainloop
  370. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 0 action 2 (virsh_node1_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=5
  371. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/35)
  372. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.6 2
  373. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.7 (null)
  374. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=7
  375. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
  376. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="virsh_node1" type="fence_virsh" class="stonith"/>
  377. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node1_last_0" operation_key="virsh_node1_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-sta
  378. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  379. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/35, version=0.6.7)
  380. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  381. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  382. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb->word_size:33792
  383. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (stonith_api_signon) debug: Connection to fencer by pacemaker-execd succeeded (registration token: d4ea23c3-5a74-4ad3-99e3-ae802a8f64a1)
  384. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:virsh_node1 action:monitor call_id:5 exit-code:7 exec-time:6ms queue-time:0ms
  385. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of probe operation for virsh_node1 on el8-a01n01: 7 (not running) | call=5 key=virsh_node1_monitor_0 confirmed=true cib-update=36
  386. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
  387. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.7 2
  388. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.8 (null)
  389. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=8
  390. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:7;1:0:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=5, @rc-code=7, @op-status=0, @last-rc-change=1594762150, @last-run=1594762150, @exec-time=6
  391. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/36, version=0.6.8)
  392. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 0 action 1 (virsh_node1_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=5
  393. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 locally on el8-a01n01 | action 3
  394. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_start_0
  395. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=6, reply=1, notify=0
  396. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:virsh_node1 action:start call_id:6
  397. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/37)
  398. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.8 2
  399. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.6.9 (null)
  400. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=9
  401. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  402. Jul 14 21:29:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/37, version=0.6.9)
  403. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
  404. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.6.9 2
  405. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.0 (null)
  406. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=7, @num_updates=0
  407. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <primitive class="stonith" id="virsh_node2" type="fence_virsh"/>
  408. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <instance_attributes id="virsh_node2-instance_attributes">
  409. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node2-instance_attributes-ipaddr" name="ipaddr" value="10.20.255.250"/>
  410. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node2-instance_attributes-login" name="login" value="root"/>
  411. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node2-instance_attributes-password_script" name="password_script" value="/root/stonith.pw"/>
  412. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n02"/>
  413. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="virsh_node2-instance_attributes-port" name="port" value="el8-a01n02"/>
  414. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </instance_attributes>
  415. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  416. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="virsh_node2-monitor-interval-60" interval="60" name="monitor"/>
  417. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  418. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  419. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  420. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.7.0
  421. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 0 aborted by primitive.virsh_node2 'create': Configuration change | cib=0.7.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
  422. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_device_register) notice: Added 'virsh_node2' to the device list (2 active devices)
  423. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.7.0)
  424. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-5.raw
  425. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.7.0 of the CIB to disk (digest: ed4816be9cba1c0fb4b22dfa9b7cb0b2)
  426. Jul 14 21:29:13 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.kG2PCG (digest: /var/lib/pacemaker/cib/cib.lhehFG)
  427. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:376837:stderr [ 2020-07-14 21:29:15,156 ERROR: Unable to connect/login to fencing device ]
  428. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:376837:stderr [ ]
  429. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:376837:stderr [ ]
  430. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[376837] stderr: [ 2020-07-14 21:29:15,156 ERROR: Unable to connect/login to fencing device ]
  431. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[376837] stderr: [ ]
  432. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[376837] stderr: [ ]
  433. Jul 14 21:29:15 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (monitor). remaining timeout is 15
  434. Jul 14 21:29:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_procfs_pid_of) info: Found pacemaker-based active as process 375409
  435. Jul 14 21:29:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (throttle_send_command) info: New throttle mode: negligible load (was undetermined)
  436. Jul 14 21:29:18 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 38bdea493108431f663dd4ee0010b3ce for 0.7.0 (0x555a63c2a960 0)
  437. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:377243:stderr [ 2020-07-14 21:29:21,392 ERROR: Unable to connect/login to fencing device ]
  438. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:377243:stderr [ ]
  439. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:377243:stderr [ ]
  440. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377243] stderr: [ 2020-07-14 21:29:21,392 ERROR: Unable to connect/login to fencing device ]
  441. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377243] stderr: [ ]
  442. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377243] stderr: [ ]
  443. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (monitor) the maximum number of times (2) allowed
  444. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) notice: Operation 'monitor' [377243] for device 'virsh_node1' returned: -201 (Generic Pacemaker error)
  445. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) info: finished - rsc:virsh_node1 action:start call_id:6 exit-code:1 exec-time:11453ms queue-time:1ms
  446. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of start operation for virsh_node1 on el8-a01n01: 1 (error) | call=6 key=virsh_node1_start_0 confirmed=true cib-update=38
  447. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/38)
  448. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.0 2
  449. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.1 (null)
  450. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=1
  451. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=6, @rc-code=1, @op-status=0, @exec-time=11453, @queue-time=1
  452. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']: <lrm_rsc_op id="virsh_node1_last_failure_0" operation_key="virsh_node1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" ca
  453. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 0 aborted by operation virsh_node1_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;3:0:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.7.1 source=process_graph_event:453 complete=false
  454. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_failcount) info: Updating failcount for virsh_node1 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762161)
  455. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/38, version=0.7.1)
  456. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) notice: Transition 0 action 3 (virsh_node1_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=6
  457. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 0 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-0.bz2): Complete
  458. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting fail-count-virsh_node1#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n01
  459. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  460. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting last-failure-virsh_node1#start_0[el8-a01n01]: (unset) -> 1594762161 | from el8-a01n01
  461. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.1 2
  462. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.2 (null)
  463. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=2
  464. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
  465. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <instance_attributes id="status-1">
  466. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="status-1-fail-count-virsh_node1.start_0" name="fail-count-virsh_node1#start_0" value="INFINITY"/>
  467. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </instance_attributes>
  468. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </transient_attributes>
  469. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/4, version=0.7.2)
  470. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.2 2
  471. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.3 (null)
  472. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=3
  473. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-virsh_node1.start_0" name="last-failure-virsh_node1#start_0" value="1594762161"/>
  474. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/5, version=0.7.3)
  475. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 0 aborted by transient_attributes.1 'create': Transient attribute change | cib=0.7.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='1'] complete=true
  476. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 0 aborted by status-1-last-failure-virsh_node1.start_0 doing create last-failure-virsh_node1#start_0=1594762161: Transient attribute change | cib=0.7.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  477. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  478. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  479. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  480. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  481. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  482. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_0
  483. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  484. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  485. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  486. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  487. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n01
  488. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  489. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n01
  490. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  491. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Recover virsh_node1 ( el8-a01n01 )
  492. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start virsh_node2 ( el8-a01n02 )
  493. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-1.bz2
  494. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_response) info: pe_calc calculation pe_calc-dc-1594762161-14 is obsolete
  495. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  496. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  497. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  498. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  499. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  500. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_0
  501. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  502. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  503. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  504. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  505. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n01
  506. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  507. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  508. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  509. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node1 on el8-a01n02
  510. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  511. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Recover virsh_node1 ( el8-a01n01 -> el8-a01n02 )
  512. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start virsh_node2 ( el8-a01n01 )
  513. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2
  514. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  515. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 2 (ref=pe_calc-dc-1594762161-15) derived from /var/lib/pacemaker/pengine/pe-input-2.bz2
  516. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 locally on el8-a01n01 | action 1
  517. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node1_stop_0
  518. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=7, reply=1, notify=0
  519. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:virsh_node1 action:stop call_id:7
  520. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) info: finished - rsc:virsh_node1 action:stop call_id:7 exit-code:0 exec-time:1ms queue-time:0ms
  521. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/42)
  522. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.3 2
  523. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.4 (null)
  524. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=4
  525. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762161, @last-run=1594762161, @exec-time=0,
  526. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/42, version=0.7.4)
  527. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n02 | action 3
  528. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n01 | action 2
  529. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'virsh_node2' not in cache
  530. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  531. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_rsc_register) info: Cached agent information for 'virsh_node2'
  532. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
  533. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  534. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.4 2
  535. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.5 (null)
  536. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=5
  537. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
  538. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-sta
  539. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  540. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/14, version=0.7.5)
  541. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_monitor_0
  542. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/43)
  543. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.5 2
  544. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.6 (null)
  545. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=6
  546. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="virsh_node2" type="fence_virsh" class="stonith"/>
  547. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="virsh_node2_last_0" operation_key="virsh_node2_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-sta
  548. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  549. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/43, version=0.7.6)
  550. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.6 2
  551. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.7 (null)
  552. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=7
  553. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;3:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=9, @rc-code=7, @op-status=0
  554. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/15, version=0.7.7)
  555. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=11, reply=1, notify=0
  556. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:virsh_node2 action:monitor call_id:11
  557. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:virsh_node2 action:monitor call_id:11 exit-code:7 exec-time:0ms queue-time:0ms
  558. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 2 action 3 (virsh_node2_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=9
  559. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok) | call=7 key=virsh_node1_stop_0 confirmed=true cib-update=44
  560. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/44)
  561. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.7 2
  562. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.8 (null)
  563. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=8
  564. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;1:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=7, @rc-code=0, @op-status=0, @exec-time=1
  565. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/44, version=0.7.8)
  566. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of probe operation for virsh_node2 on el8-a01n01: 7 (not running) | call=11 key=virsh_node2_monitor_0 confirmed=true cib-update=45
  567. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/45)
  568. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 2 action 1 (virsh_node1_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=7
  569. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.8 2
  570. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.9 (null)
  571. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=9
  572. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:7;2:2:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=11, @rc-code=7, @op-status=0
  573. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/45, version=0.7.9)
  574. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 2 action 2 (virsh_node2_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=11
  575. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation virsh_node1_start_0 on el8-a01n02 | action 4
  576. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n01 | action 6
  577. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_start_0
  578. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=12, reply=1, notify=0
  579. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:virsh_node2 action:start call_id:12
  580. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.9 2
  581. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.10 (null)
  582. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=10
  583. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_start_0, @operation=start, @transition-key=4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762161, @last-run=1594762161, @exec-time=0
  584. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/16, version=0.7.10)
  585. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/46)
  586. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.10 2
  587. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.7.11 (null)
  588. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=11
  589. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1
  590. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/46, version=0.7.11)
  591. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
  592. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.7.11 2
  593. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.8.0 (null)
  594. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=8, @num_updates=0
  595. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <primitive class="systemd" id="hypervisor" type="libvirtd"/>
  596. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  597. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-monitor-interval-60" interval="60" name="monitor"/>
  598. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-start-interval-0s" interval="0s" name="start" timeout="100"/>
  599. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  600. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  601. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  602. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.8.0)
  603. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 2 aborted by primitive.hypervisor 'create': Configuration change | cib=0.8.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
  604. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  605. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.8.0
  606. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-6.raw
  607. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.8.0 of the CIB to disk (digest: 2c9ff6874f28590448a2f242c01625bf)
  608. Jul 14 21:29:21 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8PX8sO (digest: /var/lib/pacemaker/cib/cib.YoDg1d)
  609. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2)
  610. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.8.0 2
  611. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.9.0 (null)
  612. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='hypervisor']
  613. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=9
  614. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <clone id="hypervisor-clone"/>
  615. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <primitive class="systemd" id="hypervisor" type="libvirtd">
  616. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  617. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-monitor-interval-60" interval="60" name="monitor"/>
  618. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-start-interval-0s" interval="0s" name="start" timeout="100"/>
  619. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="hypervisor-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  620. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  621. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  622. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <meta_attributes id="hypervisor-clone-meta_attributes">
  623. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="hypervisor-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
  624. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="hypervisor-clone-meta_attributes-notify" name="notify" value="false"/>
  625. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </meta_attributes>
  626. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </clone>
  627. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.9.0)
  628. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by deletion of primitive[@id='hypervisor']: Configuration change | cib=0.9.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='hypervisor'] complete=false
  629. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_device_remove) info: Device 'hypervisor' not found (2 active devices)
  630. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  631. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.9.0
  632. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-7.raw
  633. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.9.0 of the CIB to disk (digest: db64a16c52c066583a5b1a74240d556a)
  634. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AeCozH (digest: /var/lib/pacemaker/cib/cib.0ndsv8)
  635. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
  636. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.9.0 2
  637. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.10.0 (null)
  638. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=10
  639. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <primitive class="systemd" id="drbd" type="drbd"/>
  640. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  641. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-monitor-interval-60" interval="60" name="monitor"/>
  642. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-start-interval-0s" interval="0s" name="start" timeout="100"/>
  643. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  644. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  645. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  646. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by primitive.drbd 'create': Configuration change | cib=0.10.0 source=te_update_diff_v2:490 path=/cib/configuration/resources complete=false
  647. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  648. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.10.0
  649. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.10.0)
  650. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw
  651. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.10.0 of the CIB to disk (digest: 84477666f918034297fa4b47c59c0a6a)
  652. Jul 14 21:29:22 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uWgQLD (digest: /var/lib/pacemaker/cib/cib.6qhLc6)
  653. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2)
  654. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.10.0 2
  655. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.0 (null)
  656. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='drbd']
  657. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @epoch=11
  658. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/configuration/resources: <clone id="drbd-clone"/>
  659. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <primitive class="systemd" id="drbd" type="drbd">
  660. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <operations>
  661. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-monitor-interval-60" interval="60" name="monitor"/>
  662. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-start-interval-0s" interval="0s" name="start" timeout="100"/>
  663. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <op id="drbd-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  664. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </operations>
  665. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </primitive>
  666. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <meta_attributes id="drbd-clone-meta_attributes">
  667. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="drbd-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
  668. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="drbd-clone-meta_attributes-notify" name="notify" value="false"/>
  669. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </meta_attributes>
  670. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </clone>
  671. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by deletion of primitive[@id='drbd']: Configuration change | cib=0.11.0 source=te_update_diff_v2:490 path=/cib/configuration/resources/primitive[@id='drbd'] complete=false
  672. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=el8-a01n01/cibadmin/2, version=0.11.0)
  673. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_device_remove) info: Device 'drbd' not found (2 active devices)
  674. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_cib_stonith_devices_v2) info: Updating device list from the cib: create resources
  675. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-fenced [375410] (cib_devices_update) info: Updating devices to version 0.11.0
  676. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw
  677. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Wrote version 0.11.0 of the CIB to disk (digest: 2cddade26531e2c2dfcb3bac3401f84c)
  678. Jul 14 21:29:23 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.wPpOoC (digest: /var/lib/pacemaker/cib/cib.Lx5S85)
  679. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:377569:stderr [ 2020-07-14 21:29:26,751 ERROR: Unable to connect/login to fencing device ]
  680. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:377569:stderr [ ]
  681. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_1:377569:stderr [ ]
  682. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377569] stderr: [ 2020-07-14 21:29:26,751 ERROR: Unable to connect/login to fencing device ]
  683. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377569] stderr: [ ]
  684. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[377569] stderr: [ ]
  685. Jul 14 21:29:26 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (monitor). remaining timeout is 15
  686. Jul 14 21:29:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: b7814e3f7afd5c57f438342d72812427 for 0.11.0 (0x555a63c3b420 0)
  687. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.0 2
  688. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.1 (null)
  689. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=1
  690. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=10, @rc-code=1, @op-status=0, @exec-time=11419
  691. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']: <lrm_rsc_op id="virsh_node1_last_failure_0" operation_key="virsh_node1_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" ca
  692. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by operation virsh_node1_start_0 'modify' on el8-a01n02: Event failed | magic=0:1;4:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.1 source=process_graph_event:453 complete=false
  693. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_failcount) info: Updating failcount for virsh_node1 on el8-a01n02 after failed start: rc=1 (update=INFINITY, time=1594762173)
  694. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) notice: Transition 2 action 4 (virsh_node1_start_0 on el8-a01n02): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=10
  695. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting fail-count-virsh_node1#start_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
  696. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting last-failure-virsh_node1#start_0[el8-a01n02]: (unset) -> 1594762173 | from el8-a01n01
  697. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/17, version=0.11.1)
  698. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.1 2
  699. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.2 (null)
  700. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=2
  701. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/>
  702. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <instance_attributes id="status-2">
  703. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <nvpair id="status-2-fail-count-virsh_node1.start_0" name="fail-count-virsh_node1#start_0" value="INFINITY"/>
  704. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </instance_attributes>
  705. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </transient_attributes>
  706. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/6, version=0.11.2)
  707. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.11.2 source=abort_unless_down:329 path=/cib/status/node_state[@id='2'] complete=false
  708. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.2 2
  709. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.3 (null)
  710. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=3
  711. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-virsh_node1.start_0" name="last-failure-virsh_node1#start_0" value="1594762173"/>
  712. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by status-2-last-failure-virsh_node1.start_0 doing create last-failure-virsh_node1#start_0=1594762173: Transient attribute change | cib=0.11.3 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false
  713. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/7, version=0.11.3)
  714. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:378392:stderr [ 2020-07-14 21:29:32,948 ERROR: Unable to connect/login to fencing device ]
  715. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:378392:stderr [ ]
  716. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_monitor_2:378392:stderr [ ]
  717. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[378392] stderr: [ 2020-07-14 21:29:32,948 ERROR: Unable to connect/login to fencing device ]
  718. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[378392] stderr: [ ]
  719. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[378392] stderr: [ ]
  720. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (monitor) the maximum number of times (2) allowed
  721. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) notice: Operation 'monitor' [378392] for device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  722. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) info: finished - rsc:virsh_node2 action:start call_id:12 exit-code:1 exec-time:11463ms queue-time:0ms
  723. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of start operation for virsh_node2 on el8-a01n01: 1 (error) | call=12 key=virsh_node2_start_0 confirmed=true cib-update=47
  724. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/47)
  725. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.3 2
  726. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.4 (null)
  727. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=4
  728. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=12, @rc-code=1, @op-status=0, @exec-time=11463
  729. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']: <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" ca
  730. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/47, version=0.11.4)
  731. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by operation virsh_node2_start_0 'modify' on el8-a01n01: Event failed | magic=0:1;6:2:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.4 source=process_graph_event:453 complete=false
  732. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_failcount) info: Updating failcount for virsh_node2 on el8-a01n01 after failed start: rc=1 (update=INFINITY, time=1594762173)
  733. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) notice: Transition 2 action 6 (virsh_node2_start_0 on el8-a01n01): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=12
  734. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n01]: (unset) -> INFINITY | from el8-a01n01
  735. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n01]: (unset) -> 1594762173 | from el8-a01n01
  736. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 2 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete
  737. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  738. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.4 2
  739. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.5 (null)
  740. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=5
  741. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
  742. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/8, version=0.11.5)
  743. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.5 2
  744. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.6 (null)
  745. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=6
  746. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762173"/>
  747. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/9, version=0.11.6)
  748. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by status-1-fail-count-virsh_node2.start_0 doing create fail-count-virsh_node2#start_0=INFINITY: Transient attribute change | cib=0.11.5 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  749. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 2 aborted by status-1-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762173: Transient attribute change | cib=0.11.6 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
  750. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  751. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  752. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  753. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  754. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  755. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  756. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_0
  757. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  758. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_0
  759. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  760. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  761. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  762. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  763. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n02
  764. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n01
  765. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  766. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  767. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  768. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  769. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  770. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  771. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  772. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  773. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  774. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n01
  775. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:0 on el8-a01n02
  776. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:1 on el8-a01n01
  777. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n02
  778. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n01
  779. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop virsh_node1 ( el8-a01n02 ) due to node availability
  780. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Recover virsh_node2 ( el8-a01n01 )
  781. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start hypervisor:0 ( el8-a01n02 )
  782. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start hypervisor:1 ( el8-a01n01 )
  783. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start drbd:0 ( el8-a01n02 )
  784. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start drbd:1 ( el8-a01n01 )
  785. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-3.bz2
  786. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_response) info: pe_calc calculation pe_calc-dc-1594762173-21 is obsolete
  787. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  788. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  789. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  790. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  791. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  792. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  793. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_0
  794. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  795. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_0
  796. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  797. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  798. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  799. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  800. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): FAILED el8-a01n02
  801. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n01
  802. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  803. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  804. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  805. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  806. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  807. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  808. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  809. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  810. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  811. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  812. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  813. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  814. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:0 on el8-a01n01
  815. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for hypervisor:1 on el8-a01n02
  816. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for drbd:0 on el8-a01n01
  817. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for drbd:1 on el8-a01n02
  818. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop virsh_node1 ( el8-a01n02 ) due to node availability
  819. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Recover virsh_node2 ( el8-a01n01 -> el8-a01n02 )
  820. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start hypervisor:0 ( el8-a01n01 )
  821. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start hypervisor:1 ( el8-a01n02 )
  822. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start drbd:0 ( el8-a01n01 )
  823. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Start drbd:1 ( el8-a01n02 )
  824. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-4.bz2
  825. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  826. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 4 (ref=pe_calc-dc-1594762173-22) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2
  827. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation virsh_node1_stop_0 on el8-a01n02 | action 2
  828. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n01 | action 1
  829. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=virsh_node2_stop_0
  830. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/51)
  831. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=13, reply=1, notify=0
  832. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:virsh_node2 action:stop call_id:13
  833. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) info: finished - rsc:virsh_node2 action:stop call_id:13 exit-code:0 exec-time:1ms queue-time:1ms
  834. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.6 2
  835. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.7 (null)
  836. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=7
  837. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173, @exec-time=0
  838. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/51, version=0.11.7)
  839. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.7 2
  840. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.8 (null)
  841. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=8
  842. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @operation_key=virsh_node1_stop_0, @operation=stop, @transition-key=2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173, @exec-time=0
  843. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/18, version=0.11.8)
  844. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.8 2
  845. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.9 (null)
  846. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=9
  847. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node1']/lrm_rsc_op[@id='virsh_node1_last_0']: @transition-magic=0:0;2:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=11, @rc-code=0, @op-status=0, @exec-time=1
  848. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/19, version=0.11.9)
  849. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_0 locally on el8-a01n01 | action 3
  850. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor' not in cache
  851. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  852. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'hypervisor:0' not in cache
  853. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  854. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_rsc_register) info: Cached agent information for 'hypervisor'
  855. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
  856. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  857. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_monitor_0
  858. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=18, reply=1, notify=0
  859. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:18
  860. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52)
  861. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_0 on el8-a01n02 | action 5
  862. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_0 locally on el8-a01n01 | action 4
  863. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  864. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'drbd' not in cache
  865. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  866. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  867. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:18 exit-code:7 exec-time:8ms queue-time:0ms
  868. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.9 2
  869. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.10 (null)
  870. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=10
  871. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
  872. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-statu
  873. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  874. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/52, version=0.11.10)
  875. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.10 2
  876. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.11 (null)
  877. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=11
  878. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="hypervisor" type="libvirtd" class="systemd"/>
  879. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="hypervisor_last_0" operation_key="hypervisor_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-statu
  880. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  881. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/20, version=0.11.11)
  882. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_get_rsc_info) info: Agent information for 'drbd:0' not in cache
  883. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  884. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_rsc_register) info: Cached agent information for 'drbd'
  885. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_register operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=1
  886. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_info operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  887. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_monitor_0
  888. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/53)
  889. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.11 2
  890. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.12 (null)
  891. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=12
  892. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="drbd" type="drbd" class="systemd"/>
  893. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1" rc-code="193" op-status="-1" inter
  894. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  895. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/53, version=0.11.12)
  896. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=23, reply=1, notify=0
  897. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:drbd action:monitor call_id:23
  898. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  899. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  900. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.12 2
  901. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.13 (null)
  902. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=13
  903. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;5:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=16, @rc-code=7, @op-status=0, @exec-time=11
  904. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/21, version=0.11.13)
  905. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_0 on el8-a01n02 | action 6
  906. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 2 (virsh_node1_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=11
  907. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 5 (hypervisor_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=16
  908. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/54)
  909. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.13 2
  910. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.14 (null)
  911. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=14
  912. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;1:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=13, @rc-code=0, @op-status=0, @exec-time=1, @queue-time=1
  913. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/54, version=0.11.14)
  914. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:23 exit-code:7 exec-time:11ms queue-time:1ms
  915. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.14 2
  916. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.15 (null)
  917. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=15
  918. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="drbd" type="drbd" class="systemd"/>
  919. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ <lrm_rsc_op id="drbd_last_0" operation_key="drbd_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1" rc-code="193" op-status="-1" inter
  920. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ </lrm_resource>
  921. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/22, version=0.11.15)
  922. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of stop operation for virsh_node2 on el8-a01n01: 0 (ok) | call=13 key=virsh_node2_stop_0 confirmed=true cib-update=54
  923. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/55)
  924. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.15 2
  925. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.16 (null)
  926. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=16
  927. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:7;3:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=18, @rc-code=7, @op-status=0, @exec-time=8
  928. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of probe operation for hypervisor on el8-a01n01: 7 (not running) | call=18 key=hypervisor_monitor_0 confirmed=true cib-update=55
  929. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of probe operation for drbd on el8-a01n01: 7 (not running) | call=23 key=drbd_monitor_0 confirmed=true cib-update=56
  930. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/55, version=0.11.16)
  931. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/56)
  932. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 1 (virsh_node2_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=13
  933. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.16 2
  934. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.17 (null)
  935. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=17
  936. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;4:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=23, @rc-code=7, @op-status=0, @exec-time=11, @queue-time=1
  937. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 3 (hypervisor_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=18
  938. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation virsh_node2_start_0 on el8-a01n02 | action 7
  939. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation hypervisor:0_start_0 locally on el8-a01n01 | action 9
  940. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/56, version=0.11.17)
  941. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_start_0
  942. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=24, reply=1, notify=0
  943. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:hypervisor action:start call_id:24
  944. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit libvirtd named 'hypervisor'
  945. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec_with_unit) debug: Calling StartUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
  946. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation hypervisor:1_start_0 on el8-a01n02 | action 11
  947. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.17 2
  948. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.18 (null)
  949. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=18
  950. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:7;6:4:7:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=21, @rc-code=7, @op-status=0, @exec-time=14, @queue-time=1
  951. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 4 (drbd_monitor_0 on el8-a01n01) confirmed: not running | rc=7 call-id=23
  952. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/23, version=0.11.18)
  953. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 6 (drbd_monitor_0 on el8-a01n02) confirmed: not running | rc=7 call-id=21
  954. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation drbd:0_start_0 locally on el8-a01n01 | action 17
  955. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_start_0
  956. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.18 2
  957. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.19 (null)
  958. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=19
  959. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_start_0, @operation=start, @transition-key=7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762173, @last-run=1594762173
  960. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=25, reply=1, notify=0
  961. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:drbd action:start call_id:25
  962. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous start op on systemd unit drbd named 'drbd'
  963. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/24, version=0.11.19)
  964. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating start operation drbd:1_start_0 on el8-a01n02 | action 19
  965. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.19 2
  966. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.20 (null)
  967. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=20
  968. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  969. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/25, version=0.11.20)
  970. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.20 2
  971. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.21 (null)
  972. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=21
  973. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @queue-time=0
  974. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/26, version=0.11.21)
  975. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57)
  976. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58)
  977. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.21 2
  978. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.22 (null)
  979. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=22
  980. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_start_0, @operation=start, @transition-key=9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0
  981. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/57, version=0.11.22)
  982. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.22 2
  983. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.23 (null)
  984. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=23
  985. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_start_0, @operation=start, @transition-key=17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0, @queue-time=0
  986. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/58, version=0.11.23)
  987. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/4139
  988. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: hypervisor start may still be in progress: re-scheduling (elapsed=246ms, remaining=99754ms, start_delay=2000ms)
  989. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec_with_unit) debug: Calling StartUnit for drbd: /org/freedesktop/systemd1/unit/drbd_2eservice
  990. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_exec_result) info: Call to start passed: /org/freedesktop/systemd1/job/4223
  991. Jul 14 21:29:33 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: drbd start may still be in progress: re-scheduling (elapsed=506ms, remaining=99494ms, start_delay=2000ms)
  992. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.23 2
  993. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.24 (null)
  994. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=24
  995. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;11:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=23, @rc-code=0, @op-status=0, @exec-time=2222
  996. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/27, version=0.11.24)
  997. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 11 (hypervisor_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=23
  998. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation hypervisor:1_monitor_60000 on el8-a01n02 | action 12
  999. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.24 2
  1000. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.25 (null)
  1001. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=25
  1002. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']: <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a
  1003. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/28, version=0.11.25)
  1004. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.25 2
  1005. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.26 (null)
  1006. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=26
  1007. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']: @transition-magic=0:0;12:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=25, @rc-code=0, @op-status=0, @exec-time=3
  1008. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/29, version=0.11.26)
  1009. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 12 (hypervisor_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=25
  1010. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:24
  1011. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1012. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1013. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: hypervisor start is now complete (elapsed=2255ms, remaining=97745ms): ok (0)
  1014. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:24 exit-code:0 exec-time:2255ms queue-time:1ms
  1015. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of start operation for hypervisor on el8-a01n01: 0 (ok) | call=24 key=hypervisor_start_0 confirmed=true cib-update=59
  1016. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/59)
  1017. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.26 2
  1018. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.27 (null)
  1019. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=27
  1020. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;9:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=24, @rc-code=0, @op-status=0, @exec-time=2255, @queue-time=1
  1021. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/59, version=0.11.27)
  1022. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 9 (hypervisor_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=24
  1023. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation hypervisor:0_monitor_60000 locally on el8-a01n01 | action 10
  1024. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_monitor_60000
  1025. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/60)
  1026. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.27 2
  1027. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.28 (null)
  1028. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=28
  1029. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']: <lrm_rsc_op id="hypervisor_monitor_60000" operation_key="hypervisor_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a
  1030. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=26, reply=1, notify=0
  1031. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:26
  1032. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1033. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/60, version=0.11.28)
  1034. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1035. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:26 exit-code:0 exec-time:4ms queue-time:0ms
  1036. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of monitor operation for hypervisor on el8-a01n01: 0 (ok) | call=26 key=hypervisor_monitor_60000 confirmed=false cib-update=61
  1037. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/61)
  1038. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.28 2
  1039. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.29 (null)
  1040. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=29
  1041. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_monitor_60000']: @transition-magic=0:0;10:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=26, @rc-code=0, @op-status=0, @exec-time=4
  1042. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/61, version=0.11.29)
  1043. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 10 (hypervisor_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=26
  1044. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.29 2
  1045. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.30 (null)
  1046. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=30
  1047. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;19:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=24, @rc-code=0, @op-status=0, @exec-time=2409, @queue-time=1
  1048. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/30, version=0.11.30)
  1049. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 19 (drbd_start_0 on el8-a01n02) confirmed: ok | rc=0 call-id=24
  1050. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation drbd:1_monitor_60000 on el8-a01n02 | action 20
  1051. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.30 2
  1052. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.31 (null)
  1053. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=31
  1054. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="-1
  1055. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/31, version=0.11.31)
  1056. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.31 2
  1057. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.32 (null)
  1058. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=32
  1059. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;20:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=26, @rc-code=0, @op-status=0, @exec-time=3
  1060. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/32, version=0.11.32)
  1061. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 20 (drbd_monitor_60000 on el8-a01n02) confirmed: ok | rc=0 call-id=26
  1062. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:drbd action:monitor call_id:25
  1063. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1064. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1065. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: drbd start is now complete (elapsed=2511ms, remaining=97489ms): ok (0)
  1066. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:25 exit-code:0 exec-time:2511ms queue-time:1ms
  1067. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of start operation for drbd on el8-a01n01: 0 (ok) | call=25 key=drbd_start_0 confirmed=true cib-update=62
  1068. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/62)
  1069. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.32 2
  1070. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.33 (null)
  1071. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=33
  1072. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=0:0;17:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=25, @rc-code=0, @op-status=0, @exec-time=2511, @queue-time=1
  1073. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/62, version=0.11.33)
  1074. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 17 (drbd_start_0 on el8-a01n01) confirmed: ok | rc=0 call-id=25
  1075. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating monitor operation drbd:0_monitor_60000 locally on el8-a01n01 | action 18
  1076. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=drbd_monitor_60000
  1077. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/63)
  1078. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=27, reply=1, notify=0
  1079. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:drbd action:monitor call_id:27
  1080. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1081. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.33 2
  1082. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.34 (null)
  1083. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1084. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=34
  1085. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_monitor_60000" operation_key="drbd_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="-1:193;18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n01" call-id="-1
  1086. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/63, version=0.11.34)
  1087. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:3ms queue-time:0ms
  1088. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of monitor operation for drbd on el8-a01n01: 0 (ok) | call=27 key=drbd_monitor_60000 confirmed=false cib-update=64
  1089. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/64)
  1090. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.34 2
  1091. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.35 (null)
  1092. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=35
  1093. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_monitor_60000']: @transition-magic=0:0;18:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=27, @rc-code=0, @op-status=0, @exec-time=3
  1094. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/64, version=0.11.35)
  1095. Jul 14 21:29:35 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 4 action 18 (drbd_monitor_60000 on el8-a01n01) confirmed: ok | rc=0 call-id=27
  1096. Jul 14 21:29:40 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 72c0a0507a5f4ea0970661199a542caa for 0.11.35 (0x555a63c3b420 0)
  1097. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.35 2
  1098. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.36 (null)
  1099. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=36
  1100. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=22, @rc-code=1, @op-status=0, @exec-time=11492
  1101. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']: <lrm_rsc_op id="virsh_node2_last_failure_0" operation_key="virsh_node2_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" ca
  1102. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/33, version=0.11.36)
  1103. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 4 aborted by operation virsh_node2_start_0 'modify' on el8-a01n02: Event failed | magic=0:1;7:4:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.36 source=process_graph_event:453 complete=false
  1104. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_failcount) info: Updating failcount for virsh_node2 on el8-a01n02 after failed start: rc=1 (update=INFINITY, time=1594762184)
  1105. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) notice: Transition 4 action 7 (virsh_node2_start_0 on el8-a01n02): expected 'ok' but got 'error' | target-rc=0 rc=1 call-id=22
  1106. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 4 (Complete=19, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete
  1107. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1108. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting fail-count-virsh_node2#start_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
  1109. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting last-failure-virsh_node2#start_0[el8-a01n02]: (unset) -> 1594762184 | from el8-a01n01
  1110. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.36 2
  1111. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.37 (null)
  1112. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=37
  1113. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-fail-count-virsh_node2.start_0" name="fail-count-virsh_node2#start_0" value="INFINITY"/>
  1114. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/10, version=0.11.37)
  1115. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.37 2
  1116. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.38 (null)
  1117. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=38
  1118. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-virsh_node2.start_0" name="last-failure-virsh_node2#start_0" value="1594762184"/>
  1119. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/11, version=0.11.38)
  1120. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 4 aborted by status-2-fail-count-virsh_node2.start_0 doing create fail-count-virsh_node2#start_0=INFINITY: Transient attribute change | cib=0.11.37 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1121. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 4 aborted by status-2-last-failure-virsh_node2.start_0 doing create last-failure-virsh_node2#start_0=1594762184: Transient attribute change | cib=0.11.38 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1122. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  1123. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  1124. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  1125. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  1126. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1127. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1128. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1129. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1130. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_0
  1131. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1132. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1133. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1134. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1135. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1136. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n02
  1137. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1138. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1139. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1140. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1141. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1142. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1143. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1144. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1145. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1146. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1147. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1148. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (RecurringOp) info: Start recurring monitor (60s) for virsh_node2 on el8-a01n02
  1149. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1150. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Recover virsh_node2 ( el8-a01n02 )
  1151. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1152. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  1153. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1154. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  1155. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-5.bz2
  1156. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_response) info: pe_calc calculation pe_calc-dc-1594762184-38 is obsolete
  1157. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  1158. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  1159. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n02 is active
  1160. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is online
  1161. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1162. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1163. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1164. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1165. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_0
  1166. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1167. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1168. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1169. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1170. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1171. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): FAILED el8-a01n02
  1172. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1173. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1174. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1175. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1176. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1177. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1178. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1179. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1180. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1181. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1182. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1183. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1184. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1185. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1186. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1187. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop virsh_node2 ( el8-a01n02 ) due to node availability
  1188. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1189. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Started el8-a01n02)
  1190. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1191. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave drbd:1 (Started el8-a01n02)
  1192. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-6.bz2
  1193. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1194. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 6 (ref=pe_calc-dc-1594762184-39) derived from /var/lib/pacemaker/pengine/pe-input-6.bz2
  1195. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation virsh_node2_stop_0 on el8-a01n02 | action 3
  1196. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.38 2
  1197. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.39 (null)
  1198. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=39
  1199. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @operation_key=virsh_node2_stop_0, @operation=stop, @transition-key=3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762184, @last-run=1594762184, @exec-time=0
  1200. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/34, version=0.11.39)
  1201. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.39 2
  1202. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.40 (null)
  1203. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=40
  1204. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virsh_node2']/lrm_rsc_op[@id='virsh_node2_last_0']: @transition-magic=0:0;3:6:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=27, @rc-code=0, @op-status=0, @exec-time=1
  1205. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/35, version=0.11.40)
  1206. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 6 action 3 (virsh_node2_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=27
  1207. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 6 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-6.bz2): Complete
  1208. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
  1209. Jul 14 21:29:44 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
  1210. Jul 14 21:29:49 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 8251df6e54b59aa0e832ca3aef06d2cb for 0.11.40 (0x555a63c3b420 0)
  1211. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_expected) info: handle_request: Node el8-a01n02[2] - expected state is now down (was member)
  1212. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_shutdown_request) info: Creating shutdown request for el8-a01n02 (state=S_IDLE)
  1213. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting shutdown[el8-a01n02]: (unset) -> 1594762228 | from el8-a01n01
  1214. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.40 2
  1215. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.41 (null)
  1216. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=41
  1217. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-shutdown" name="shutdown" value="1594762228"/>
  1218. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/12, version=0.11.41)
  1219. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 6 aborted by status-2-shutdown doing create shutdown=1594762228: Transient attribute change | cib=0.11.41 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1220. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
  1221. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status_fencing) info: Node el8-a01n01 is active
  1222. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is online
  1223. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1224. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1225. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1226. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1227. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1228. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1229. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1230. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1231. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1232. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1233. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1234. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1235. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1236. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1237. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 el8-a01n02 ]
  1238. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1239. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1240. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1241. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1242. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1243. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1244. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1245. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1246. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1247. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1248. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1249. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1250. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n02
  1251. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n02
  1252. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1253. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1254. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Started el8-a01n01)
  1255. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop hypervisor:1 ( el8-a01n02 ) due to node availability
  1256. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave drbd:0 (Started el8-a01n01)
  1257. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1258. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) notice: Calculated transition 7, saving inputs in /var/lib/pacemaker/pengine/pe-input-7.bz2
  1259. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1260. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 7 (ref=pe_calc-dc-1594762228-41) derived from /var/lib/pacemaker/pengine/pe-input-7.bz2
  1261. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 on el8-a01n02 | action 7
  1262. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation drbd_stop_0 on el8-a01n02 | action 14
  1263. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.41 2
  1264. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.42 (null)
  1265. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=42
  1266. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762228, @last-run=1594762228, @exec-time=0
  1267. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/36, version=0.11.42)
  1268. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.42 2
  1269. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.43 (null)
  1270. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=43
  1271. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @operation_key=drbd_stop_0, @operation=stop, @transition-key=14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762228, @last-run=1594762228, @exec-time=0, @queue-time=0
  1272. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/37, version=0.11.43)
  1273. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd [375401] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  1274. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd [375401] (pcmk_shutdown_worker) notice: Shutting down Pacemaker
  1275. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemakerd [375401] (stop_child) notice: Stopping pacemaker-controld | sent signal 15 to process 375414
  1276. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler)
  1277. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_shutdown) notice: Shutting down cluster resource manager | limit=1200000ms
  1278. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_log) warning: Input I_SHUTDOWN received in state S_TRANSITION_ENGINE from crm_shutdown
  1279. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown
  1280. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_shutdown_req) info: Sending shutdown request to all peers (DC is el8-a01n01)
  1281. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (crm_update_peer_expected) info: handle_request: Node el8-a01n01[1] - expected state is now down (was member)
  1282. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_shutdown_request) info: Creating shutdown request for el8-a01n01 (state=S_POLICY_ENGINE)
  1283. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting shutdown[el8-a01n01]: (unset) -> 1594762228 | from el8-a01n01
  1284. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.43 2
  1285. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.44 (null)
  1286. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=44
  1287. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-shutdown" name="shutdown" value="1594762228"/>
  1288. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/13, version=0.11.44)
  1289. Jul 14 21:30:28 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 7 aborted by status-1-shutdown doing create shutdown=1594762228: Transient attribute change | cib=0.11.44 source=abort_unless_down:329 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false
  1290. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.44 2
  1291. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.45 (null)
  1292. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=45
  1293. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;7:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2224
  1294. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/38, version=0.11.45)
  1295. Jul 14 21:30:30 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 7 action 7 (hypervisor_stop_0 on el8-a01n02) confirmed: ok | rc=0 call-id=29
  1296. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1297. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1298. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1299. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:26 exit-code:0 exec-time:0ms queue-time:0ms
  1300. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: afb079b332593e47dff6eff041966396 for 0.11.45 (0x555a63c3b420 0)
  1301. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1302. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1303. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1304. Jul 14 21:30:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:0ms queue-time:0ms
  1305. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (recurring_action_timer) debug: Scheduling another invocation of hypervisor_status_60000
  1306. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1307. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1308. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:26 exit-code:0 exec-time:0ms queue-time:0ms
  1309. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1310. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1311. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1312. Jul 14 21:31:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:0ms queue-time:0ms
  1313. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.45 2
  1314. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.46 (null)
  1315. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=46
  1316. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']/lrm_rsc_op[@id='drbd_last_0']: @transition-magic=2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=31, @rc-code=198, @op-status=2, @exec-time=99991
  1317. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='drbd']: <lrm_rsc_op id="drbd_last_failure_0" operation_key="drbd_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.3.0" transition-key="14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" transition-magic="2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11" exit-reason="" on_node="el8-a01n02" call-id="31" rc-code=
  1318. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/crmd/39, version=0.11.46)
  1319. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 7 aborted by operation drbd_stop_0 'modify' on el8-a01n02: Event failed | magic=2:198;14:7:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 cib=0.11.46 source=process_graph_event:453 complete=false
  1320. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (update_failcount) info: Updating failcount for drbd on el8-a01n02 after failed stop: rc=198 (update=INFINITY, time=1594762328)
  1321. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY | from el8-a01n01
  1322. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-attrd [375412] (attrd_peer_update) notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594762328 | from el8-a01n01
  1323. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) notice: Transition 7 action 14 (drbd_stop_0 on el8-a01n02): expected 'ok' but got 'OCF_TIMEOUT' | target-rc=0 rc=198 call-id=31
  1324. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 7 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-7.bz2): Complete
  1325. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.46 2
  1326. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.47 (null)
  1327. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=47
  1328. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-fail-count-drbd.stop_0" name="fail-count-drbd#stop_0" value="INFINITY"/>
  1329. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/14, version=0.11.47)
  1330. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 7 aborted by status-2-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change | cib=0.11.47 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1331. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.47 2
  1332. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.48 (null)
  1333. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=48
  1334. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-last-failure-drbd.stop_0" name="last-failure-drbd#stop_0" value="1594762328"/>
  1335. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n02/attrd/15, version=0.11.48)
  1336. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) info: Transition 7 aborted by status-2-last-failure-drbd.stop_0 doing create last-failure-drbd#stop_0=1594762328: Transient attribute change | cib=0.11.48 source=abort_unless_down:329 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true
  1337. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1338. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1339. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1340. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1341. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1342. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1343. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1344. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1345. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1346. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1347. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1348. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1349. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1350. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1351. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1352. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1353. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1354. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n02 ]
  1355. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1356. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1357. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1358. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1359. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1360. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1361. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1362. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1363. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1364. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1365. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1366. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1367. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1368. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1369. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1370. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1371. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1372. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1373. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1374. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1375. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1376. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1377. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1378. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1379. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1380. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1381. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1382. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1383. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability
  1384. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1385. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1386. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1387. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 8 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-0.bz2
  1388. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (handle_response) info: pe_calc calculation pe_calc-dc-1594762328-45 is obsolete
  1389. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1390. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1391. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1392. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1393. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1394. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1395. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1396. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1397. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1398. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1399. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1400. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1401. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1402. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1403. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1404. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1405. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1406. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1407. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n02 ]
  1408. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1409. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1410. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1411. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1412. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1413. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1414. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1415. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1416. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1417. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1418. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1419. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1420. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1421. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1422. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1423. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1424. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1425. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1426. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1427. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1428. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1429. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1430. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1431. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1432. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1433. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1434. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1435. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1436. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop hypervisor:0 ( el8-a01n01 ) due to node availability
  1437. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1438. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1439. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1440. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 9 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-0.bz2
  1441. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1442. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 9 (ref=pe_calc-dc-1594762328-46) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
  1443. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=4 timeout=60000
  1444. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_rsc_command) notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n01 | action 5
  1445. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1446. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=2bbcc655-c155-4908-a792-c918b427a38e state=0
  1447. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (cancel_recurring_action) info: Cancelling systemd operation hypervisor_status_60000
  1448. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:26 exit-code:0 exec-time:0ms queue-time:0ms
  1449. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_cancel operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=0, reply=1, notify=0
  1450. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1451. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1452. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 2bbcc655-c155-4908-a792-c918b427a38e
  1453. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 2bbcc655-c155-4908-a792-c918b427a38e
  1454. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1455. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_lrm_rsc_op) info: Performing key=5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11 op=hypervisor_stop_0
  1456. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=2bbcc655-c155-4908-a792-c918b427a38e
  1457. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/72)
  1458. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1459. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (process_lrmd_message) debug: Processed lrmd_rsc_exec operation from 09acd970-a9ad-4616-9dfb-40686968c8db: rc=29, reply=1, notify=0
  1460. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) info: executing - rsc:hypervisor action:stop call_id:29
  1461. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous stop op on systemd unit libvirtd named 'hypervisor'
  1462. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.48 2
  1463. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.49 (null)
  1464. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=49
  1465. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @operation_key=hypervisor_stop_0, @operation=stop, @transition-key=5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @transition-magic=-1:193;5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1594762328, @last-run=1594762328, @exec-time=0, @qu
  1466. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/72, version=0.11.49)
  1467. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) info: Result of monitor operation for hypervisor on el8-a01n01: Cancelled | call=26 key=hypervisor_monitor_60000 confirmed=true
  1468. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec_with_unit) debug: Calling StopUnit for hypervisor: /org/freedesktop/systemd1/unit/libvirtd_2eservice
  1469. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1470. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1471. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1472. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_exec_result) info: Call to stop passed: /org/freedesktop/systemd1/job/4328
  1473. Jul 14 21:32:08 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: hypervisor stop may still be in progress: re-scheduling (elapsed=267ms, remaining=99733ms, start_delay=2000ms)
  1474. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_execute) debug: executing - rsc:hypervisor action:monitor call_id:29
  1475. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit libvirtd named 'hypervisor'
  1476. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1477. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (action_complete) debug: hypervisor stop is now complete (elapsed=2271ms, remaining=97729ms): not running (7)
  1478. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:hypervisor action:monitor call_id:29 exit-code:0 exec-time:2271ms queue-time:0ms
  1479. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/73)
  1480. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_lrm_event) notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok) | call=29 key=hypervisor_stop_0 confirmed=true cib-update=73
  1481. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: --- 0.11.49 2
  1482. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: Diff: +++ 0.11.50 (null)
  1483. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib: @num_updates=50
  1484. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='hypervisor']/lrm_rsc_op[@id='hypervisor_last_0']: @transition-magic=0:0;5:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11, @call-id=29, @rc-code=0, @op-status=0, @exec-time=2271
  1485. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=el8-a01n01/crmd/73, version=0.11.50)
  1486. Jul 14 21:32:10 el8-a01n01.digimer.ca pacemaker-controld [375414] (process_graph_event) info: Transition 9 action 5 (hypervisor_stop_0 on el8-a01n01) confirmed: ok | rc=0 call-id=29
  1487. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:388864:stderr [ 2020-07-14 21:32:13,484 ERROR: Unable to connect/login to fencing device ]
  1488. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:388864:stderr [ ]
  1489. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:388864:stderr [ ]
  1490. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[388864] stderr: [ 2020-07-14 21:32:13,484 ERROR: Unable to connect/login to fencing device ]
  1491. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[388864] stderr: [ ]
  1492. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[388864] stderr: [ ]
  1493. Jul 14 21:32:13 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
  1494. Jul 14 21:32:15 el8-a01n01.digimer.ca pacemaker-based [375409] (cib_process_ping) info: Reporting our current digest to el8-a01n01: 55204ae50dead0b498c548c5d466607c for 0.11.50 (0x555a63c3b420 0)
  1495. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:389235:stderr [ 2020-07-14 21:32:19,741 ERROR: Unable to connect/login to fencing device ]
  1496. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:389235:stderr [ ]
  1497. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:389235:stderr [ ]
  1498. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389235] stderr: [ 2020-07-14 21:32:19,741 ERROR: Unable to connect/login to fencing device ]
  1499. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389235] stderr: [ ]
  1500. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389235] stderr: [ ]
  1501. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1502. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [389235] (call 2 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1503. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1504. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1505. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.2bbcc655: No route to host
  1506. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 2/4:9:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1507. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 2 for el8-a01n02 failed (No route to host): aborting transition.
  1508. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 9 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1509. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=2bbcc655-c155-4908-a792-c918b427a38e
  1510. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 9 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Complete
  1511. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1512. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1513. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1514. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1515. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1516. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1517. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1518. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1519. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1520. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1521. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1522. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1523. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1524. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1525. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1526. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1527. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1528. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1529. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1530. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1531. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1532. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1533. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1534. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1535. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1536. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1537. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1538. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1539. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1540. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1541. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1542. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1543. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1544. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1545. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1546. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1547. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1548. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1549. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1550. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1551. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1552. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1553. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1554. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1555. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1556. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1557. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  1558. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1559. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1560. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1561. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 10 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1562. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1563. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 10 (ref=pe_calc-dc-1594762339-48) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1564. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  1565. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1566. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=75793d25-0284-4916-823f-ebadcf09771b state=0
  1567. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1568. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1569. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 75793d25-0284-4916-823f-ebadcf09771b
  1570. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 75793d25-0284-4916-823f-ebadcf09771b
  1571. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1572. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=75793d25-0284-4916-823f-ebadcf09771b
  1573. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1574. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1575. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1576. Jul 14 21:32:19 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1577. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:389538:stderr [ 2020-07-14 21:32:24,971 ERROR: Unable to connect/login to fencing device ]
  1578. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:389538:stderr [ ]
  1579. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:389538:stderr [ ]
  1580. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389538] stderr: [ 2020-07-14 21:32:24,971 ERROR: Unable to connect/login to fencing device ]
  1581. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389538] stderr: [ ]
  1582. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[389538] stderr: [ ]
  1583. Jul 14 21:32:25 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
  1584. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390256:stderr [ 2020-07-14 21:32:31,205 ERROR: Unable to connect/login to fencing device ]
  1585. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390256:stderr [ ]
  1586. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390256:stderr [ ]
  1587. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390256] stderr: [ 2020-07-14 21:32:31,205 ERROR: Unable to connect/login to fencing device ]
  1588. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390256] stderr: [ ]
  1589. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390256] stderr: [ ]
  1590. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1591. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [390256] (call 3 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1592. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1593. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1594. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.75793d25: No route to host
  1595. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 3/3:10:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1596. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 3 for el8-a01n02 failed (No route to host): aborting transition.
  1597. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 10 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1598. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=75793d25-0284-4916-823f-ebadcf09771b
  1599. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 10 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
  1600. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1601. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1602. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1603. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1604. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1605. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1606. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1607. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1608. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1609. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1610. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1611. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1612. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1613. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1614. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1615. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1616. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1617. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1618. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1619. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1620. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1621. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1622. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1623. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1624. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1625. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1626. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1627. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1628. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1629. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1630. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1631. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1632. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1633. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1634. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1635. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1636. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1637. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1638. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1639. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1640. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1641. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1642. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1643. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1644. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1645. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1646. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1647. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  1648. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1649. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1650. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1651. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 11 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1652. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1653. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 11 (ref=pe_calc-dc-1594762351-49) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1654. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  1655. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1656. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=5be1f6ba-66c0-4d35-920f-a083bf59809c state=0
  1657. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1658. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1659. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 5be1f6ba-66c0-4d35-920f-a083bf59809c
  1660. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 5be1f6ba-66c0-4d35-920f-a083bf59809c
  1661. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1662. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=5be1f6ba-66c0-4d35-920f-a083bf59809c
  1663. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1664. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1665. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1666. Jul 14 21:32:31 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1667. Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (recurring_action_timer) debug: Scheduling another invocation of drbd_status_60000
  1668. Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (systemd_unit_exec) debug: Performing asynchronous status op on systemd unit drbd named 'drbd'
  1669. Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (pcmk_dbus_get_property) debug: Calling: GetAll on org.freedesktop.systemd1
  1670. Jul 14 21:32:35 el8-a01n01.digimer.ca pacemaker-execd [375411] (log_finished) debug: finished - rsc:drbd action:monitor call_id:27 exit-code:0 exec-time:0ms queue-time:0ms
  1671. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:390569:stderr [ 2020-07-14 21:32:36,434 ERROR: Unable to connect/login to fencing device ]
  1672. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:390569:stderr [ ]
  1673. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:390569:stderr [ ]
  1674. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390569] stderr: [ 2020-07-14 21:32:36,434 ERROR: Unable to connect/login to fencing device ]
  1675. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390569] stderr: [ ]
  1676. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390569] stderr: [ ]
  1677. Jul 14 21:32:36 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
  1678. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390879:stderr [ 2020-07-14 21:32:42,776 ERROR: Unable to connect/login to fencing device ]
  1679. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390879:stderr [ ]
  1680. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:390879:stderr [ ]
  1681. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390879] stderr: [ 2020-07-14 21:32:42,776 ERROR: Unable to connect/login to fencing device ]
  1682. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390879] stderr: [ ]
  1683. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[390879] stderr: [ ]
  1684. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1685. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [390879] (call 4 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1686. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1687. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1688. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.5be1f6ba: No route to host
  1689. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 4/3:11:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1690. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 4 for el8-a01n02 failed (No route to host): aborting transition.
  1691. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 11 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1692. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=5be1f6ba-66c0-4d35-920f-a083bf59809c
  1693. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 11 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
  1694. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1695. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1696. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1697. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1698. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1699. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1700. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1701. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1702. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1703. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1704. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1705. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1706. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1707. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1708. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1709. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1710. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1711. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1712. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1713. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1714. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1715. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1716. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1717. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1718. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1719. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1720. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1721. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1722. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1723. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1724. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1725. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1726. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1727. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1728. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1729. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1730. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1731. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1732. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1733. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1734. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1735. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1736. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1737. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1738. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1739. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1740. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1741. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  1742. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1743. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1744. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1745. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 12 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1746. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1747. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 12 (ref=pe_calc-dc-1594762362-50) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1748. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  1749. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1750. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=9fc8382b-26df-4e45-8f59-affe54245144 state=0
  1751. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1752. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1753. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 9fc8382b-26df-4e45-8f59-affe54245144
  1754. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 9fc8382b-26df-4e45-8f59-affe54245144
  1755. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1756. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=9fc8382b-26df-4e45-8f59-affe54245144
  1757. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1758. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1759. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1760. Jul 14 21:32:42 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1761. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391149:stderr [ 2020-07-14 21:32:48,173 ERROR: Unable to connect/login to fencing device ]
  1762. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391149:stderr [ ]
  1763. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391149:stderr [ ]
  1764. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391149] stderr: [ 2020-07-14 21:32:48,173 ERROR: Unable to connect/login to fencing device ]
  1765. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391149] stderr: [ ]
  1766. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391149] stderr: [ ]
  1767. Jul 14 21:32:48 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
  1768. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:391399:stderr [ 2020-07-14 21:32:54,383 ERROR: Unable to connect/login to fencing device ]
  1769. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:391399:stderr [ ]
  1770. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:391399:stderr [ ]
  1771. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391399] stderr: [ 2020-07-14 21:32:54,383 ERROR: Unable to connect/login to fencing device ]
  1772. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391399] stderr: [ ]
  1773. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391399] stderr: [ ]
  1774. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1775. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [391399] (call 5 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1776. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1777. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1778. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.9fc8382b: No route to host
  1779. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 5/3:12:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1780. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 5 for el8-a01n02 failed (No route to host): aborting transition.
  1781. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 12 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1782. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=9fc8382b-26df-4e45-8f59-affe54245144
  1783. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 12 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
  1784. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1785. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1786. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1787. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1788. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1789. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1790. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1791. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1792. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1793. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1794. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1795. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1796. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1797. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1798. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1799. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1800. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1801. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1802. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1803. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1804. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1805. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1806. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1807. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1808. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1809. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1810. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1811. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1812. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1813. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1814. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1815. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1816. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1817. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1818. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1819. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1820. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1821. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1822. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1823. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1824. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1825. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1826. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1827. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1828. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1829. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1830. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1831. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  1832. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1833. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1834. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1835. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 13 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1836. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1837. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 13 (ref=pe_calc-dc-1594762374-51) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1838. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  1839. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1840. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=d3fc9798-6df8-485b-a7d3-e904d314cf81 state=0
  1841. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1842. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1843. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) d3fc9798-6df8-485b-a7d3-e904d314cf81
  1844. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) d3fc9798-6df8-485b-a7d3-e904d314cf81
  1845. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1846. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=d3fc9798-6df8-485b-a7d3-e904d314cf81
  1847. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1848. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1849. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1850. Jul 14 21:32:54 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1851. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391738:stderr [ 2020-07-14 21:32:59,589 ERROR: Unable to connect/login to fencing device ]
  1852. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391738:stderr [ ]
  1853. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:391738:stderr [ ]
  1854. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391738] stderr: [ 2020-07-14 21:32:59,589 ERROR: Unable to connect/login to fencing device ]
  1855. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391738] stderr: [ ]
  1856. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[391738] stderr: [ ]
  1857. Jul 14 21:32:59 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 55
  1858. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392102:stderr [ 2020-07-14 21:33:05,784 ERROR: Unable to connect/login to fencing device ]
  1859. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392102:stderr [ ]
  1860. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392102:stderr [ ]
  1861. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392102] stderr: [ 2020-07-14 21:33:05,784 ERROR: Unable to connect/login to fencing device ]
  1862. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392102] stderr: [ ]
  1863. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392102] stderr: [ ]
  1864. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1865. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [392102] (call 6 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1866. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1867. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1868. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.d3fc9798: No route to host
  1869. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 6/3:13:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1870. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 6 for el8-a01n02 failed (No route to host): aborting transition.
  1871. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 13 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1872. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=d3fc9798-6df8-485b-a7d3-e904d314cf81
  1873. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 13 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
  1874. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1875. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1876. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1877. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1878. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1879. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1880. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1881. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1882. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1883. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1884. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1885. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1886. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1887. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1888. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1889. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1890. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1891. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1892. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1893. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1894. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1895. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1896. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1897. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1898. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1899. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1900. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1901. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1902. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1903. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1904. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1905. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1906. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1907. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1908. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1909. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  1910. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  1911. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  1912. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  1913. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  1914. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  1915. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  1916. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  1917. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  1918. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  1919. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  1920. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  1921. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  1922. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  1923. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  1924. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  1925. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 14 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1926. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  1927. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 14 (ref=pe_calc-dc-1594762385-52) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  1928. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  1929. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  1930. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=3e12455e-82d1-4460-90db-9dcb37d6de0e state=0
  1931. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1932. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1933. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 3e12455e-82d1-4460-90db-9dcb37d6de0e
  1934. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 3e12455e-82d1-4460-90db-9dcb37d6de0e
  1935. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  1936. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=3e12455e-82d1-4460-90db-9dcb37d6de0e
  1937. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  1938. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  1939. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  1940. Jul 14 21:33:05 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
  1941. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:392399:stderr [ 2020-07-14 21:33:11,068 ERROR: Unable to connect/login to fencing device ]
  1942. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:392399:stderr [ ]
  1943. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_1:392399:stderr [ ]
  1944. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392399] stderr: [ 2020-07-14 21:33:11,068 ERROR: Unable to connect/login to fencing device ]
  1945. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392399] stderr: [ ]
  1946. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392399] stderr: [ ]
  1947. Jul 14 21:33:11 el8-a01n01.digimer.ca pacemaker-fenced [375410] (internal_stonith_action_execute) info: Attempt 2 to execute fence_virsh (reboot). remaining timeout is 54
  1948. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392682:stderr [ 2020-07-14 21:33:17,444 ERROR: Unable to connect/login to fencing device ]
  1949. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392682:stderr [ ]
  1950. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (operation_finished) notice: fence_virsh_reboot_2:392682:stderr [ ]
  1951. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392682] stderr: [ 2020-07-14 21:33:17,444 ERROR: Unable to connect/login to fencing device ]
  1952. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392682] stderr: [ ]
  1953. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_action) warning: fence_virsh[392682] stderr: [ ]
  1954. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (update_remaining_timeout) info: Attempted to execute agent fence_virsh (reboot) the maximum number of times (2) allowed
  1955. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (log_operation) error: Operation 'reboot' [392682] (call 7 from pacemaker-controld.375414) for host 'el8-a01n02' with device 'virsh_node2' returned: -201 (Generic Pacemaker error)
  1956. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_choose_peer) notice: Couldn't find anyone to fence (reboot) el8-a01n02 with any device
  1957. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: No peers (out of 2) are capable of fencing (reboot) el8-a01n02 for client pacemaker-controld.375414 | state=1
  1958. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (remote_op_done) error: Operation 'reboot' targeting el8-a01n02 on <no-one> for pacemaker-controld.375414@el8-a01n01.3e12455e: No route to host
  1959. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 7/3:14:0:0df6240b-f9a0-4c7b-bf60-fd2dab145e11: No route to host (-113)
  1960. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_callback) notice: Stonith operation 7 for el8-a01n02 failed (No route to host): aborting transition.
  1961. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (abort_transition_graph) notice: Transition 14 aborted: Stonith failed | source=abort_for_stonith_failure:256 complete=false
  1962. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (tengine_stonith_notify) notice: Peer el8-a01n02 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.375414: No route to host | initiator=el8-a01n01 ref=3e12455e-82d1-4460-90db-9dcb37d6de0e
  1963. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (run_graph) notice: Transition 14 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
  1964. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
  1965. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (process_pe_message) info: Input has not changed since last time, not saving to disk
  1966. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n01 is shutting down
  1967. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (determine_online_status) info: Node el8-a01n02 is shutting down
  1968. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n01 at Jul 14 21:29:10 2020 | rc=1 id=virsh_node1_last_failure_0
  1969. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n01 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node2_last_failure_0
  1970. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node1 on el8-a01n02 at Jul 14 21:29:21 2020 | rc=1 id=virsh_node1_last_failure_0
  1971. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of virsh_node2 on el8-a01n02 at Jul 14 21:29:33 2020 | rc=1 id=virsh_node2_last_failure_0
  1972. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_failure_0
  1973. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_rsc_op_failure) warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n02 at Jul 14 21:30:28 2020 | rc=198 id=drbd_last_0
  1974. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_fence_node) warning: Cluster node el8-a01n02 will be fenced: drbd:1 failed there
  1975. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1976. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1977. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 1 is already processed
  1978. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (unpack_node_loop) info: Node 2 is already processed
  1979. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node1 (stonith:fence_virsh): Stopped
  1980. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: virsh_node2 (stonith:fence_virsh): Stopped
  1981. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: hypervisor-clone [hypervisor]
  1982. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Stopped: [ el8-a01n01 el8-a01n02 ]
  1983. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (clone_print) info: Clone Set: drbd-clone [drbd]
  1984. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (common_print) info: drbd (systemd:drbd): FAILED el8-a01n02
  1985. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (short_print) info: Started: [ el8-a01n01 ]
  1986. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n01
  1987. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n01 after 1000000 failures (max=1000000)
  1988. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n01
  1989. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n01 after 1000000 failures (max=1000000)
  1990. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node1 has failed INFINITY times on el8-a01n02
  1991. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node1 away from el8-a01n02 after 1000000 failures (max=1000000)
  1992. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: virsh_node2 has failed INFINITY times on el8-a01n02
  1993. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing virsh_node2 away from el8-a01n02 after 1000000 failures (max=1000000)
  1994. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:0 has failed INFINITY times on el8-a01n02
  1995. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1996. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pe_get_failcount) info: drbd:1 has failed INFINITY times on el8-a01n02
  1997. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (check_migration_threshold) warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  1998. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node1 cannot run anywhere
  1999. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource virsh_node2 cannot run anywhere
  2000. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:0 cannot run anywhere
  2001. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource hypervisor:1 cannot run anywhere
  2002. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:0 cannot run anywhere
  2003. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_color) info: Resource drbd:1 cannot run anywhere
  2004. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (sched_shutdown_op) notice: Scheduling shutdown of node el8-a01n01
  2005. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (stage6) warning: Scheduling Node el8-a01n02 for STONITH
  2006. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (native_stop_constraints) notice: Stop of failed resource drbd:1 is implicit after el8-a01n02 is fenced
  2007. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Shutdown el8-a01n01
  2008. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogNodeActions) notice: * Fence (reboot) el8-a01n02 'drbd:1 failed there'
  2009. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node1 (Stopped)
  2010. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave virsh_node2 (Stopped)
  2011. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:0 (Stopped)
  2012. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogActions) info: Leave hypervisor:1 (Stopped)
  2013. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:0 ( el8-a01n01 ) due to node availability
  2014. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (LogAction) notice: * Stop drbd:1 ( el8-a01n02 ) due to node availability
  2015. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-schedulerd[375413] (pcmk__log_transition_summary) warning: Calculated transition 15 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-1.bz2
  2016. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
  2017. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (do_te_invoke) info: Processing graph 15 (ref=pe_calc-dc-1594762397-53) derived from /var/lib/pacemaker/pengine/pe-warn-1.bz2
  2018. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-controld [375414] (te_fence_node) notice: Requesting fencing (reboot) of node el8-a01n02 | action=3 timeout=60000
  2019. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (handle_request) notice: Client pacemaker-controld.375414.cb1de721 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  2020. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (initiate_remote_stonith_op) notice: Requesting peer fencing (reboot) targeting el8-a01n02 | id=97afe994-2cc3-45d0-887f-c94bed0d27e1 state=0
  2021. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  2022. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  2023. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 1 of 2 from el8-a01n01 for el8-a01n02/reboot (1 devices) 97afe994-2cc3-45d0-887f-c94bed0d27e1
  2024. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: Query result 2 of 2 from el8-a01n02 for el8-a01n02/reboot (0 devices) 97afe994-2cc3-45d0-887f-c94bed0d27e1
  2025. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (process_remote_stonith_query) info: All query replies have arrived, continuing (2 expected/2 received)
  2026. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) info: Total timeout set to 60 for peer's fencing targeting el8-a01n02 for pacemaker-controld.375414|id=97afe994-2cc3-45d0-887f-c94bed0d27e1
  2027. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (call_remote_stonith) notice: Requesting that el8-a01n01 perform 'reboot' action targeting el8-a01n02 | for client pacemaker-controld.375414 (72s, 0s)
  2028. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  2029. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (can_fence_host_with_device) notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  2030. Jul 14 21:33:17 el8-a01n01.digimer.ca pacemaker-fenced [375410] (stonith_fence_get_devices_cb) info: Found 1 matching devices for 'el8-a01n02'
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!