API tools faq
paste
digimer

Untitled

digimer
Jul 14th, 2020
138
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 61.74 KB | None | 0 0
raw download clone embed print report
  1. [root@el8-a01n01 ~]# dnf list installed | grep drbd
  2. drbd90-utils.x86_64 9.13.1-1.el8 @el8-striker01-repo
  3. kmod-drbd90.x86_64 9.0.23-1.el8 @el8-striker01-repo
  4.  
  5. [root@el8-a01n01 ~]# systemctl status drbd.service
  6. ● drbd.service - DRBD -- please disable. Unless you are NOT using a cluster manager.
  7. Loaded: loaded (/usr/lib/systemd/system/drbd.service; disabled; vendor preset: disabled)
  8. Active: inactive (dead)
  9.  
  10. [root@el8-a01n01 ~]# systemctl start drbd.service
  11.  
  12. Jul 14 05:02:59 el8-a01n01.digimer.ca systemd[1]: Starting DRBD -- please disable. Unless you are NOT using a cluster manager....
  13. Jul 14 05:02:59 el8-a01n01.digimer.ca kernel: drbd: loading out-of-tree module taints kernel.
  14. Jul 14 05:02:59 el8-a01n01.digimer.ca kernel: drbd: module verification failed: signature and/or required key missing - tainting kernel
  15. Jul 14 05:02:59 el8-a01n01.digimer.ca kernel: drbd: initialized. Version: 9.0.23-1 (api:2/proto:86-116)
  16. Jul 14 05:02:59 el8-a01n01.digimer.ca kernel: drbd: GIT-hash: d16bfab7a4033024fed2d99d3b179aa6bb6eb300 build by admin@el8-b3.alteeve.com, 2020-06-08 17:04:59
  17. Jul 14 05:02:59 el8-a01n01.digimer.ca kernel: drbd: registered as block device major 147
  18. Jul 14 05:02:59 el8-a01n01.digimer.ca drbd[2430]: Starting DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such file or directory
  19. Jul 14 05:02:59 el8-a01n01.digimer.ca drbd[2430]: no resources defined!
  20. Jul 14 05:02:59 el8-a01n01.digimer.ca drbd[2430]: no resources defined!
  21. Jul 14 05:02:59 el8-a01n01.digimer.ca drbd[2430]: WARN: stdin/stdout is not a TTY; using /dev/consoleWARN: stdin/stdout is not a TTY; using /dev/console.
  22. Jul 14 05:02:59 el8-a01n01.digimer.ca systemd[1]: Started DRBD -- please disable. Unless you are NOT using a cluster manager..
  23. Jul 14 05:03:01 el8-a01n01.digimer.ca systemd[1]: Started Session 61 of user root.
  24.  
  25. [root@el8-a01n01 ~]# systemctl stop drbd.service
  26.  
  27. Jul 14 05:03:47 el8-a01n01.digimer.ca systemd[1]: Stopping DRBD -- please disable. Unless you are NOT using a cluster manager....
  28. Jul 14 05:03:47 el8-a01n01.digimer.ca drbd[2461]: Stopping all DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such file or directory
  29. Jul 14 05:03:47 el8-a01n01.digimer.ca kernel: drbd: module cleanup done.
  30. Jul 14 05:03:47 el8-a01n01.digimer.ca drbd[2461]: .
  31. Jul 14 05:03:47 el8-a01n01.digimer.ca systemd[1]: Stopped DRBD -- please disable. Unless you are NOT using a cluster manager..
  32.  
  33. [root@el8-a01n01 ~]# cat /var/lib/pacemaker/cib/cib.xml
  34.  
  35. <cib crm_feature_set="3.3.0" validate-with="pacemaker-3.2" epoch="14" num_updates="0" admin_epoch="0" cib-last-written="Tue Jul 14 04:29:57 2020" update-origin="el8-a01n01" update-client="cibadmin" update-user="root" have-quorum="1" dc-uuid="2">
  36. <configuration>
  37. <crm_config>
  38. <cluster_property_set id="cib-bootstrap-options">
  39. <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
  40. <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.0.3-5.el8_2.1-4b1f869f0f"/>
  41. <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
  42. <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="el8-anvil-01"/>
  43. <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/>
  44. </cluster_property_set>
  45. </crm_config>
  46. <nodes>
  47. <node id="1" uname="el8-a01n01"/>
  48. <node id="2" uname="el8-a01n02"/>
  49. </nodes>
  50. <resources>
  51. <primitive class="stonith" id="virsh_node1" type="fence_virsh">
  52. <instance_attributes id="virsh_node1-instance_attributes">
  53. <nvpair id="virsh_node1-instance_attributes-delay" name="delay" value="15"/>
  54. <nvpair id="virsh_node1-instance_attributes-ipaddr" name="ipaddr" value="192.168.122.1"/>
  55. <nvpair id="virsh_node1-instance_attributes-login" name="login" value="root"/>
  56. <nvpair id="virsh_node1-instance_attributes-passwd" name="passwd" value="shallow decrease purple"/>
  57. <nvpair id="virsh_node1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n01"/>
  58. <nvpair id="virsh_node1-instance_attributes-port" name="port" value="el8-a01n01"/>
  59. </instance_attributes>
  60. <operations>
  61. <op id="virsh_node1-monitor-interval-60" interval="60" name="monitor"/>
  62. </operations>
  63. </primitive>
  64. <primitive class="stonith" id="virsh_node2" type="fence_virsh">
  65. <instance_attributes id="virsh_node2-instance_attributes">
  66. <nvpair id="virsh_node2-instance_attributes-ipaddr" name="ipaddr" value="192.168.122.1"/>
  67. <nvpair id="virsh_node2-instance_attributes-login" name="login" value="root"/>
  68. <nvpair id="virsh_node2-instance_attributes-passwd" name="passwd" value="shallow decrease purple"/>
  69. <nvpair id="virsh_node2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="el8-a01n02"/>
  70. <nvpair id="virsh_node2-instance_attributes-port" name="port" value="el8-a01n02"/>
  71. </instance_attributes>
  72. <operations>
  73. <op id="virsh_node2-monitor-interval-60" interval="60" name="monitor"/>
  74. </operations>
  75. </primitive>
  76. <clone id="hypervisor-clone">
  77. <primitive class="systemd" id="hypervisor" type="libvirtd">
  78. <operations>
  79. <op id="hypervisor-monitor-interval-60" interval="60" name="monitor"/>
  80. <op id="hypervisor-start-interval-0s" interval="0s" name="start" timeout="100"/>
  81. <op id="hypervisor-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  82. </operations>
  83. </primitive>
  84. <meta_attributes id="hypervisor-clone-meta_attributes">
  85. <nvpair id="hypervisor-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
  86. <nvpair id="hypervisor-clone-meta_attributes-notify" name="notify" value="false"/>
  87. </meta_attributes>
  88. </clone>
  89. <clone id="drbd-clone">
  90. <primitive class="systemd" id="drbd" type="drbd">
  91. <operations>
  92. <op id="drbd-monitor-interval-60" interval="60" name="monitor"/>
  93. <op id="drbd-start-interval-0s" interval="0s" name="start" timeout="100"/>
  94. <op id="drbd-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
  95. </operations>
  96. </primitive>
  97. <meta_attributes id="drbd-clone-meta_attributes">
  98. <nvpair id="drbd-clone-meta_attributes-clone-max" name="clone-max" value="2"/>
  99. <nvpair id="drbd-clone-meta_attributes-notify" name="notify" value="false"/>
  100. </meta_attributes>
  101. </clone>
  102. </resources>
  103. <constraints/>
  104. </configuration>
  105. </cib>
  106.  
  107. [root@el8-a01n01 ~]# systemctl status drbd.service
  108. ● drbd.service - DRBD -- please disable. Unless you are NOT using a cluster manager.
  109. Loaded: loaded (/usr/lib/systemd/system/drbd.service; disabled; vendor preset: disabled)
  110. Active: inactive (dead)
  111.  
  112. Jul 14 05:06:00 el8-a01n01.digimer.ca systemd[1]: Starting DRBD -- please disable. Unless you are NOT using a cluster manager....
  113. Jul 14 05:06:00 el8-a01n01.digimer.ca drbd[2513]: Starting DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such f>
  114. Jul 14 05:06:00 el8-a01n01.digimer.ca drbd[2513]: no resources defined!
  115. Jul 14 05:06:00 el8-a01n01.digimer.ca drbd[2513]: no resources defined!
  116. Jul 14 05:06:00 el8-a01n01.digimer.ca drbd[2513]: WARN: stdin/stdout is not a TTY; using /dev/consoleWARN: stdin/stdout is not a TTY; using /dev/con>
  117. Jul 14 05:06:00 el8-a01n01.digimer.ca systemd[1]: Started DRBD -- please disable. Unless you are NOT using a cluster manager..
  118. Jul 14 05:06:34 el8-a01n01.digimer.ca systemd[1]: Stopping DRBD -- please disable. Unless you are NOT using a cluster manager....
  119. Jul 14 05:06:34 el8-a01n01.digimer.ca drbd[2633]: Stopping all DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No su>
  120. Jul 14 05:06:34 el8-a01n01.digimer.ca drbd[2633]: .
  121. Jul 14 05:06:34 el8-a01n01.digimer.ca systemd[1]: Stopped DRBD -- please disable. Unless you are NOT using a cluster manager..
  122.  
  123. [root@el8-a01n01 ~]# pcs cluster start --all
  124. el8-a01n01: Starting Cluster...
  125. el8-a01n02: Starting Cluster...
  126.  
  127. Jul 14 05:08:01 el8-a01n01.digimer.ca systemd[1]: Started Session 66 of user root.
  128. Jul 14 05:08:01 el8-a01n01.digimer.ca CROND[2693]: (root) CMD (/usr/sbin/anvil-update-issue >> /var/log/anvil.cron 2>&1 )
  129. Jul 14 05:08:03 el8-a01n01.digimer.ca pcsd[1138]: Thin::Server
  130. Jul 14 05:08:03 el8-a01n01.digimer.ca pcsd[1138]: Thin web server (v1.7.2 codename Bachmanity)
  131. Jul 14 05:08:03 el8-a01n01.digimer.ca pcsd[1138]: Maximum connections set to 1024
  132. Jul 14 05:08:03 el8-a01n01.digimer.ca pcsd[1138]: Listening on /run/pcsd-ruby.socket, CTRL+C to stop
  133. Jul 14 05:08:03 el8-a01n01.digimer.ca systemd[1]: Starting Corosync Cluster Engine...
  134. Jul 14 05:08:03 el8-a01n01.digimer.ca corosync[2712]: [MAIN ] Corosync Cluster Engine 3.0.3 starting up
  135. Jul 14 05:08:03 el8-a01n01.digimer.ca corosync[2712]: [MAIN ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow
  136. Jul 14 05:08:03 el8-a01n01.digimer.ca corosync[2712]: [TOTEM ] Initializing transport (Kronosnet).
  137. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [TOTEM ] kronosnet crypto initialized: aes256/sha256
  138. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [TOTEM ] totemknet initialized
  139. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so
  140. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync configuration map access [0]
  141. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QB ] server name: cmap
  142. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync configuration service [1]
  143. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QB ] server name: cfg
  144. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
  145. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QB ] server name: cpg
  146. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync profile loading service [4]
  147. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QUORUM] Using quorum provider corosync_votequorum
  148. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  149. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
  150. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QB ] server name: votequorum
  151. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
  152. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QB ] server name: quorum
  153. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 0)
  154. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 1 has no active links
  155. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
  156. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 has no active links
  157. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
  158. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 has no active links
  159. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
  160. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 has no active links
  161. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [TOTEM ] A new membership (1.57) was formed. Members joined: 1
  162. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  163. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [CPG ] downlist left_list: 0 received
  164. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  165. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  166. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [QUORUM] Members[1]: 1
  167. Jul 14 05:08:04 el8-a01n01.digimer.ca corosync[2712]: [MAIN ] Completed service synchronization, ready to provide service.
  168. Jul 14 05:08:04 el8-a01n01.digimer.ca systemd[1]: Started Corosync Cluster Engine.
  169. Jul 14 05:08:04 el8-a01n01.digimer.ca systemd[1]: Reached target resource-agents dependencies.
  170. Jul 14 05:08:04 el8-a01n01.digimer.ca systemd[1]: Started Pacemaker High Availability Cluster Manager.
  171. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  172. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Starting Pacemaker 2.0.3-5.el8_2.1
  173. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemakerd[2724]: warning: Quorum lost
  174. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Pacemaker daemon successfully started and accepting connections
  175. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Node el8-a01n01 state is now member
  176. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  177. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-execd[2727]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  178. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Starting Pacemaker fencer
  179. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Connecting to cluster infrastructure: corosync
  180. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-execd[2727]: notice: Starting Pacemaker local executor
  181. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-execd[2727]: notice: Pacemaker local executor successfully started and accepting connections
  182. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  183. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Starting Pacemaker CIB manager
  184. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-schedulerd[2729]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  185. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-schedulerd[2729]: notice: Starting Pacemaker scheduler
  186. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-schedulerd[2729]: notice: Pacemaker scheduler successfully started and accepting connections
  187. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  188. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Starting Pacemaker node attribute manager
  189. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  190. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Starting Pacemaker controller
  191. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Node el8-a01n01 state is now member
  192. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Connecting to cluster infrastructure: corosync
  193. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Node el8-a01n01 state is now member
  194. Jul 14 05:08:04 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Pacemaker CIB manager successfully started and accepting connections
  195. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [KNET ] rx: host: 2 link: 0 is up
  196. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
  197. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
  198. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [KNET ] pmtud: Global data MTU changed to: 1397
  199. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Connecting to cluster infrastructure: corosync
  200. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Connecting to cluster infrastructure: corosync
  201. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [TOTEM ] A new membership (1.5b) was formed. Members joined: 2
  202. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [CPG ] downlist left_list: 0 received
  203. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [CPG ] downlist left_list: 0 received
  204. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  205. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [QUORUM] This node is within the primary component and will provide service.
  206. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [QUORUM] Members[2]: 1 2
  207. Jul 14 05:08:05 el8-a01n01.digimer.ca corosync[2712]: [MAIN ] Completed service synchronization, ready to provide service.
  208. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Quorum acquired
  209. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Node el8-a01n02 state is now member
  210. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-based[2725]: notice: Node el8-a01n02 state is now member
  211. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Pacemaker fencer successfully started and accepting connections
  212. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Node el8-a01n01 state is now member
  213. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Node el8-a01n02 state is now member
  214. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Pacemaker node attribute manager successfully started and accepting connections
  215. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Quorum acquired
  216. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Node el8-a01n02 state is now member
  217. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2
  218. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2
  219. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Node el8-a01n01 state is now member
  220. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Node el8-a01n02 state is now member
  221. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Pacemaker controller successfully started and accepting connections
  222. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: State transition S_STARTING -> S_PENDING
  223. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Added 'virsh_node1' to the device list (1 active devices)
  224. Jul 14 05:08:05 el8-a01n01.digimer.ca pacemaker-fenced[2726]: notice: Added 'virsh_node2' to the device list (2 active devices)
  225. Jul 14 05:08:06 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Fencer successfully connected
  226. Jul 14 05:08:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: State transition S_PENDING -> S_NOT_DC
  227. Jul 14 05:08:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of probe operation for virsh_node1 on el8-a01n01: 7 (not running)
  228. Jul 14 05:08:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of probe operation for virsh_node2 on el8-a01n01: 7 (not running)
  229. Jul 14 05:08:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of probe operation for hypervisor on el8-a01n01: 7 (not running)
  230. Jul 14 05:08:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of probe operation for drbd on el8-a01n01: 7 (not running)
  231. Jul 14 05:08:26 el8-a01n01.digimer.ca systemd[1]: Reloading.
  232. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Listening on Virtual machine log manager socket.
  233. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Listening on Virtual machine lock manager socket.
  234. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Created slice Virtual Machine and Container Slice.
  235. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Starting Virtual Machine and Container Registration Service...
  236. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Started Virtual Machine and Container Registration Service.
  237. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Starting Cluster Controlled libvirtd...
  238. Jul 14 05:08:27 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of start operation for virsh_node1 on el8-a01n01: 0 (ok)
  239. Jul 14 05:08:27 el8-a01n01.digimer.ca systemd[1]: Started Cluster Controlled libvirtd.
  240. Jul 14 05:08:28 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of monitor operation for virsh_node1 on el8-a01n01: 0 (ok)
  241. Jul 14 05:08:28 el8-a01n01.digimer.ca systemd[1]: Reloading.
  242. Jul 14 05:08:28 el8-a01n01.digimer.ca systemd[1]: Starting Cluster Controlled drbd...
  243. Jul 14 05:08:28 el8-a01n01.digimer.ca kernel: drbd: initialized. Version: 9.0.23-1 (api:2/proto:86-116)
  244. Jul 14 05:08:28 el8-a01n01.digimer.ca kernel: drbd: GIT-hash: d16bfab7a4033024fed2d99d3b179aa6bb6eb300 build by admin@el8-b3.alteeve.com, 2020-06-08 17:04:59
  245. Jul 14 05:08:28 el8-a01n01.digimer.ca kernel: drbd: registered as block device major 147
  246. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: Starting DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such file or directory
  247. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: no resources defined!
  248. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: no resources defined!
  249. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: WARN: stdin/stdout is not a TTY; using /dev/consoleWARN: stdin/stdout is not a TTY; using /dev/console.
  250. Jul 14 05:08:28 el8-a01n01.digimer.ca systemd[1]: Started Cluster Controlled drbd.
  251. Jul 14 05:08:29 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of start operation for hypervisor on el8-a01n01: 0 (ok)
  252. Jul 14 05:08:29 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of monitor operation for hypervisor on el8-a01n01: 0 (ok)
  253. Jul 14 05:08:30 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of start operation for drbd on el8-a01n01: 0 (ok)
  254. Jul 14 05:08:30 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of monitor operation for drbd on el8-a01n01: 0 (ok)
  255. ====
  256. Jul 14 05:08:01 el8-a01n02.digimer.ca systemd[1]: Started Session 66 of user root.
  257. Jul 14 05:08:01 el8-a01n02.digimer.ca CROND[2557]: (root) CMD (/usr/sbin/anvil-update-issue >> /var/log/anvil.cron 2>&1 )
  258. Jul 14 05:08:03 el8-a01n02.digimer.ca pcsd[1046]: Thin::Server
  259. Jul 14 05:08:03 el8-a01n02.digimer.ca pcsd[1046]: Thin web server (v1.7.2 codename Bachmanity)
  260. Jul 14 05:08:03 el8-a01n02.digimer.ca pcsd[1046]: Maximum connections set to 1024
  261. Jul 14 05:08:03 el8-a01n02.digimer.ca pcsd[1046]: Listening on /run/pcsd-ruby.socket, CTRL+C to stop
  262. Jul 14 05:08:03 el8-a01n02.digimer.ca systemd[1]: Starting Corosync Cluster Engine...
  263. Jul 14 05:08:03 el8-a01n02.digimer.ca corosync[2570]: [MAIN ] Corosync Cluster Engine 3.0.3 starting up
  264. Jul 14 05:08:03 el8-a01n02.digimer.ca corosync[2570]: [MAIN ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow
  265. Jul 14 05:08:03 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] Initializing transport (Kronosnet).
  266. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] kronosnet crypto initialized: aes256/sha256
  267. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] totemknet initialized
  268. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so
  269. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync configuration map access [0]
  270. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QB ] server name: cmap
  271. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync configuration service [1]
  272. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QB ] server name: cfg
  273. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
  274. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QB ] server name: cpg
  275. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync profile loading service [4]
  276. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QUORUM] Using quorum provider corosync_votequorum
  277. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  278. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
  279. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QB ] server name: votequorum
  280. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
  281. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QB ] server name: quorum
  282. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 0)
  283. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 has no active links
  284. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
  285. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 has no active links
  286. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
  287. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 has no active links
  288. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] A new membership (2.57) was formed. Members joined: 2
  289. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  290. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [CPG ] downlist left_list: 0 received
  291. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  292. Jul 14 05:08:04 el8-a01n02.digimer.ca systemd[1]: Started Corosync Cluster Engine.
  293. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
  294. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [QUORUM] Members[1]: 2
  295. Jul 14 05:08:04 el8-a01n02.digimer.ca corosync[2570]: [MAIN ] Completed service synchronization, ready to provide service.
  296. Jul 14 05:08:04 el8-a01n02.digimer.ca systemd[1]: Reached target resource-agents dependencies.
  297. Jul 14 05:08:04 el8-a01n02.digimer.ca systemd[1]: Started Pacemaker High Availability Cluster Manager.
  298. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  299. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Starting Pacemaker 2.0.3-5.el8_2.1
  300. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemakerd[2582]: warning: Quorum lost
  301. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Pacemaker daemon successfully started and accepting connections
  302. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Node el8-a01n02 state is now member
  303. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  304. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Starting Pacemaker CIB manager
  305. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-execd[2585]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  306. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  307. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  308. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Starting Pacemaker node attribute manager
  309. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-execd[2585]: notice: Starting Pacemaker local executor
  310. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-execd[2585]: notice: Pacemaker local executor successfully started and accepting connections
  311. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  312. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Starting Pacemaker scheduler
  313. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Pacemaker scheduler successfully started and accepting connections
  314. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
  315. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Starting Pacemaker controller
  316. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Starting Pacemaker fencer
  317. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Connecting to cluster infrastructure: corosync
  318. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Connecting to cluster infrastructure: corosync
  319. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Node el8-a01n02 state is now member
  320. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Node el8-a01n02 state is now member
  321. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Pacemaker CIB manager successfully started and accepting connections
  322. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Pacemaker fencer successfully started and accepting connections
  323. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Added 'virsh_node1' to the device list (1 active devices)
  324. Jul 14 05:08:04 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Added 'virsh_node2' to the device list (2 active devices)
  325. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [KNET ] rx: host: 1 link: 0 is up
  326. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
  327. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [KNET ] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 1397
  328. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [KNET ] pmtud: Global data MTU changed to: 1397
  329. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Connecting to cluster infrastructure: corosync
  330. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Connecting to cluster infrastructure: corosync
  331. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] A new membership (1.5b) was formed. Members joined: 1
  332. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [CPG ] downlist left_list: 0 received
  333. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [CPG ] downlist left_list: 0 received
  334. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [QUORUM] This node is within the primary component and will provide service.
  335. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [QUORUM] Members[2]: 1 2
  336. Jul 14 05:08:05 el8-a01n02.digimer.ca corosync[2570]: [MAIN ] Completed service synchronization, ready to provide service.
  337. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Quorum acquired
  338. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Node el8-a01n01 state is now member
  339. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Node el8-a01n02 state is now member
  340. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Node el8-a01n01 state is now member
  341. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Node el8-a01n01 state is now member
  342. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Pacemaker node attribute manager successfully started and accepting connections
  343. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Node el8-a01n01 state is now member
  344. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting #attrd-protocol[el8-a01n02]: (unset) -> 2
  345. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Quorum acquired
  346. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting #attrd-protocol[el8-a01n01]: (unset) -> 2
  347. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Node el8-a01n01 state is now member
  348. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Node el8-a01n02 state is now member
  349. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Pacemaker controller successfully started and accepting connections
  350. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: State transition S_STARTING -> S_PENDING
  351. Jul 14 05:08:05 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Recorded local node as attribute writer (was unset)
  352. Jul 14 05:08:06 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Fencer successfully connected
  353. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: warning: Input I_DC_TIMEOUT received in state S_PENDING from crm_timer_popped
  354. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: State transition S_ELECTION -> S_INTEGRATION
  355. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start virsh_node1 ( el8-a01n01 )
  356. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start virsh_node2 ( el8-a01n02 )
  357. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start hypervisor:0 ( el8-a01n01 )
  358. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start hypervisor:1 ( el8-a01n02 )
  359. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start drbd:0 ( el8-a01n01 )
  360. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Start drbd:1 ( el8-a01n02 )
  361. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-9.bz2
  362. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node1_monitor_0 locally on el8-a01n02
  363. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node1_monitor_0 on el8-a01n01
  364. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node2_monitor_0 locally on el8-a01n02
  365. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node2_monitor_0 on el8-a01n01
  366. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of probe operation for virsh_node1 on el8-a01n02: 7 (not running)
  367. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of probe operation for virsh_node2 on el8-a01n02: 7 (not running)
  368. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation hypervisor:1_monitor_0 locally on el8-a01n02
  369. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation drbd:1_monitor_0 locally on el8-a01n02
  370. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of probe operation for hypervisor on el8-a01n02: 7 (not running)
  371. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of probe operation for drbd on el8-a01n02: 7 (not running)
  372. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation virsh_node1_start_0 on el8-a01n01
  373. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation virsh_node2_start_0 locally on el8-a01n02
  374. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation hypervisor:0_monitor_0 on el8-a01n01
  375. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation drbd:0_monitor_0 on el8-a01n01
  376. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation hypervisor:1_start_0 locally on el8-a01n02
  377. Jul 14 05:08:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation hypervisor:0_start_0 on el8-a01n01
  378. Jul 14 05:08:26 el8-a01n02.digimer.ca systemd[1]: Reloading.
  379. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Created slice Virtual Machine and Container Slice.
  380. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Listening on Virtual machine lock manager socket.
  381. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Starting Virtual Machine and Container Registration Service...
  382. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Listening on Virtual machine log manager socket.
  383. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Started Virtual Machine and Container Registration Service.
  384. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Starting Cluster Controlled libvirtd...
  385. Jul 14 05:08:27 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of start operation for virsh_node2 on el8-a01n02: 0 (ok)
  386. Jul 14 05:08:27 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node2_monitor_60000 locally on el8-a01n02
  387. Jul 14 05:08:27 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation virsh_node1_monitor_60000 on el8-a01n01
  388. Jul 14 05:08:27 el8-a01n02.digimer.ca systemd[1]: Started Cluster Controlled libvirtd.
  389. Jul 14 05:08:28 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of monitor operation for virsh_node2 on el8-a01n02: 0 (ok)
  390. Jul 14 05:08:28 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation drbd:1_start_0 locally on el8-a01n02
  391. Jul 14 05:08:28 el8-a01n02.digimer.ca systemd[1]: Reloading.
  392. Jul 14 05:08:28 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating start operation drbd:0_start_0 on el8-a01n01
  393. Jul 14 05:08:28 el8-a01n02.digimer.ca systemd[1]: Starting Cluster Controlled drbd...
  394. Jul 14 05:08:28 el8-a01n02.digimer.ca kernel: drbd: initialized. Version: 9.0.23-1 (api:2/proto:86-116)
  395. Jul 14 05:08:28 el8-a01n02.digimer.ca kernel: drbd: GIT-hash: d16bfab7a4033024fed2d99d3b179aa6bb6eb300 build by admin@el8-b3.alteeve.com, 2020-06-08 17:04:59
  396. Jul 14 05:08:28 el8-a01n02.digimer.ca kernel: drbd: registered as block device major 147
  397. Jul 14 05:08:28 el8-a01n02.digimer.ca drbd[2834]: Starting DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such file or directory
  398. Jul 14 05:08:28 el8-a01n02.digimer.ca drbd[2834]: no resources defined!
  399. Jul 14 05:08:28 el8-a01n02.digimer.ca drbd[2834]: no resources defined!
  400. Jul 14 05:08:28 el8-a01n02.digimer.ca drbd[2834]: WARN: stdin/stdout is not a TTY; using /dev/consoleWARN: stdin/stdout is not a TTY; using /dev/console.
  401. Jul 14 05:08:28 el8-a01n02.digimer.ca systemd[1]: Started Cluster Controlled drbd.
  402. Jul 14 05:08:29 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation hypervisor:0_monitor_60000 on el8-a01n01
  403. Jul 14 05:08:29 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of start operation for hypervisor on el8-a01n02: 0 (ok)
  404. Jul 14 05:08:29 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation hypervisor:1_monitor_60000 locally on el8-a01n02
  405. Jul 14 05:08:29 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of monitor operation for hypervisor on el8-a01n02: 0 (ok)
  406. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of start operation for drbd on el8-a01n02: 0 (ok)
  407. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation drbd:1_monitor_60000 locally on el8-a01n02
  408. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating monitor operation drbd:0_monitor_60000 on el8-a01n01
  409. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of monitor operation for drbd on el8-a01n02: 0 (ok)
  410. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 0 (Complete=24, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-9.bz2): Complete
  411. Jul 14 05:08:30 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
  412.  
  413. [root@el8-a01n01 ~]# pcs cluster start --all
  414. el8-a01n01: Starting Cluster...
  415. el8-a01n02: Starting Cluster...
  416. [root@el8-a01n01 ~]# systemctl status drbd.service
  417. ● drbd.service - Cluster Controlled drbd
  418. Loaded: loaded (/usr/lib/systemd/system/drbd.service; disabled; vendor preset: disabled)
  419. Drop-In: /run/systemd/system/drbd.service.d
  420. └─50-pacemaker.conf
  421. Active: active (exited) since Tue 2020-07-14 05:08:28 GMT; 48s ago
  422. Process: 2975 ExecStart=/lib/drbd/drbd start (code=exited, status=0/SUCCESS)
  423. Main PID: 2975 (code=exited, status=0/SUCCESS)
  424.  
  425. Jul 14 05:08:28 el8-a01n01.digimer.ca systemd[1]: Starting Cluster Controlled drbd...
  426. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: Starting DRBD resources: /lib/drbd/drbd: line 148: /var/lib/linstor/loop_device_mapping: No such f>
  427. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: no resources defined!
  428. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: no resources defined!
  429. Jul 14 05:08:28 el8-a01n01.digimer.ca drbd[2975]: WARN: stdin/stdout is not a TTY; using /dev/consoleWARN: stdin/stdout is not a TTY; using /dev/con>
  430. Jul 14 05:08:28 el8-a01n01.digimer.ca systemd[1]: Started Cluster Controlled drbd.
  431.  
  432. [root@el8-a01n01 ~]# pcs status
  433. Cluster name: el8-anvil-01
  434. Cluster Summary:
  435. * Stack: corosync
  436. * Current DC: el8-a01n02 (version 2.0.3-5.el8_2.1-4b1f869f0f) - partition with quorum
  437. * Last updated: Tue Jul 14 05:08:39 2020
  438. * Last change: Tue Jul 14 04:29:57 2020 by root via cibadmin on el8-a01n01
  439. * 2 nodes configured
  440. * 6 resource instances configured
  441.  
  442. Node List:
  443. * Online: [ el8-a01n01 el8-a01n02 ]
  444.  
  445. Full List of Resources:
  446. * virsh_node1 (stonith:fence_virsh): Started el8-a01n01
  447. * virsh_node2 (stonith:fence_virsh): Started el8-a01n02
  448. * Clone Set: hypervisor-clone [hypervisor]:
  449. * Started: [ el8-a01n01 el8-a01n02 ]
  450. * Clone Set: drbd-clone [drbd]:
  451. * Started: [ el8-a01n01 el8-a01n02 ]
  452.  
  453. Daemon Status:
  454. corosync: active/disabled
  455. pacemaker: active/disabled
  456. pcsd: active/enabled
  457.  
  458. [root@el8-a01n01 ~]# pcs cluster stop --all
  459. # (doesn't return)
  460.  
  461. Jul 14 05:11:22 el8-a01n01.digimer.ca systemd[1]: Stopping Pacemaker High Availability Cluster Manager...
  462. Jul 14 05:11:22 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Caught 'Terminated' signal
  463. Jul 14 05:11:22 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Shutting down Pacemaker
  464. Jul 14 05:11:22 el8-a01n01.digimer.ca pacemakerd[2724]: notice: Stopping pacemaker-controld
  465. Jul 14 05:11:22 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Caught 'Terminated' signal
  466. Jul 14 05:11:22 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Shutting down cluster resource manager
  467. Jul 14 05:11:23 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting shutdown[el8-a01n01]: (unset) -> 1594703482
  468. Jul 14 05:11:23 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting shutdown[el8-a01n02]: (unset) -> 1594703483
  469. Jul 14 05:11:23 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of stop operation for virsh_node1 on el8-a01n01: 0 (ok)
  470. Jul 14 05:11:23 el8-a01n01.digimer.ca systemd[1]: Reloading.
  471. Jul 14 05:11:24 el8-a01n01.digimer.ca systemd[1]: Stopping Virtualization daemon...
  472. Jul 14 05:11:24 el8-a01n01.digimer.ca systemd[1]: Reloading.
  473. Jul 14 05:11:24 el8-a01n01.digimer.ca systemd[1]: Stopped Virtualization daemon.
  474. Jul 14 05:11:26 el8-a01n01.digimer.ca pacemaker-controld[2730]: notice: Result of stop operation for hypervisor on el8-a01n01: 0 (ok)
  475. ====
  476. Jul 14 05:11:22 el8-a01n02.digimer.ca systemd[1]: Stopping Pacemaker High Availability Cluster Manager...
  477. Jul 14 05:11:22 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Caught 'Terminated' signal
  478. Jul 14 05:11:22 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Shutting down Pacemaker
  479. Jul 14 05:11:22 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Stopping pacemaker-controld
  480. Jul 14 05:11:22 el8-a01n02.digimer.ca pacemaker-attrd[2586]: error: Error in connection setup (/dev/shm/qb-2586-2588-10-6N3Xle/qb): Broken pipe (32)
  481. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Caught 'Terminated' signal
  482. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Shutting down cluster resource manager
  483. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: State transition S_IDLE -> S_POLICY_ENGINE
  484. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting shutdown[el8-a01n01]: (unset) -> 1594703482
  485. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting shutdown[el8-a01n02]: (unset) -> 1594703483
  486. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Scheduling shutdown of node el8-a01n01
  487. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Shutdown el8-a01n01
  488. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Move virsh_node1 ( el8-a01n01 -> el8-a01n02 )
  489. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop hypervisor:1 ( el8-a01n01 ) due to node availability
  490. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:1 ( el8-a01n01 ) due to node availability
  491. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-10.bz2
  492. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Scheduling shutdown of node el8-a01n01
  493. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Scheduling shutdown of node el8-a01n02
  494. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Shutdown el8-a01n02
  495. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Shutdown el8-a01n01
  496. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop virsh_node1 ( el8-a01n01 ) due to node availability
  497. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop virsh_node2 ( el8-a01n02 ) due to node availability
  498. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop hypervisor:0 ( el8-a01n02 ) due to node availability
  499. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop hypervisor:1 ( el8-a01n01 ) due to node availability
  500. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:0 ( el8-a01n02 ) due to node availability
  501. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:1 ( el8-a01n01 ) due to node availability
  502. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-11.bz2
  503. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation virsh_node1_stop_0 on el8-a01n01
  504. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation virsh_node2_stop_0 locally on el8-a01n02
  505. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of stop operation for virsh_node2 on el8-a01n02: 0 (ok)
  506. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation hypervisor_stop_0 locally on el8-a01n02
  507. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation hypervisor_stop_0 on el8-a01n01
  508. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation drbd_stop_0 locally on el8-a01n02
  509. Jul 14 05:11:23 el8-a01n02.digimer.ca systemd[1]: Reloading.
  510. Jul 14 05:11:23 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Initiating stop operation drbd_stop_0 on el8-a01n01
  511. Jul 14 05:11:24 el8-a01n02.digimer.ca systemd[1]: Stopping Virtualization daemon...
  512. Jul 14 05:11:24 el8-a01n02.digimer.ca systemd[1]: Reloading.
  513. Jul 14 05:11:24 el8-a01n02.digimer.ca systemd[1]: Stopped Virtualization daemon.
  514. Jul 14 05:11:26 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Result of stop operation for hypervisor on el8-a01n02: 0 (ok)
  515.  
  516. ### After a delay
  517.  
  518. Jul 14 05:13:01 el8-a01n01.digimer.ca systemd[1]: Started Session 71 of user root.
  519. Jul 14 05:13:01 el8-a01n01.digimer.ca CROND[3211]: (root) CMD (/usr/sbin/anvil-update-issue >> /var/log/anvil.cron 2>&1 )
  520. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-execd[2727]: notice: Giving up on drbd stop (rc=0): timeout (elapsed=99980ms, remaining=20ms)
  521. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-controld[2730]: error: Result of stop operation for drbd on el8-a01n01: Timed Out
  522. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY
  523. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594703583
  524. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting fail-count-drbd#stop_0[el8-a01n01]: (unset) -> INFINITY
  525. Jul 14 05:13:03 el8-a01n01.digimer.ca pacemaker-attrd[2728]: notice: Setting last-failure-drbd#stop_0[el8-a01n01]: (unset) -> 1594703583
  526. ====
  527. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-execd[2585]: notice: Giving up on drbd stop (rc=0): timeout (elapsed=99992ms, remaining=8ms)
  528. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: error: Result of stop operation for drbd on el8-a01n02: Timed Out
  529. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 2 aborted by operation drbd_stop_0 'modify' on el8-a01n02: Event failed
  530. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 2 action 15 (drbd_stop_0 on el8-a01n02): expected 'ok' but got 'OCF_TIMEOUT'
  531. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting fail-count-drbd#stop_0[el8-a01n02]: (unset) -> INFINITY
  532. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting last-failure-drbd#stop_0[el8-a01n02]: (unset) -> 1594703583
  533. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 2 aborted by status-2-fail-count-drbd.stop_0 doing create fail-count-drbd#stop_0=INFINITY: Transient attribute change
  534. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 2 action 16 (drbd_stop_0 on el8-a01n01): expected 'ok' but got 'OCF_TIMEOUT'
  535. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Transition 2 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-11.bz2): Complete
  536. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting fail-count-drbd#stop_0[el8-a01n01]: (unset) -> INFINITY
  537. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Setting last-failure-drbd#stop_0[el8-a01n01]: (unset) -> 1594703583
  538. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n02 at Jul 14 05:11:23 2020
  539. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n02 at Jul 14 05:11:23 2020
  540. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Cluster node el8-a01n02 will be fenced: drbd:0 failed there
  541. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n01 at Jul 14 05:11:23 2020
  542. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n01 at Jul 14 05:11:23 2020
  543. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Cluster node el8-a01n01 will be fenced: drbd:1 failed there
  544. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  545. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  546. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Scheduling Node el8-a01n01 for STONITH
  547. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Stop of failed resource drbd:1 is implicit after el8-a01n01 is fenced
  548. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Scheduling Node el8-a01n02 for STONITH
  549. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Stop of failed resource drbd:0 is implicit after el8-a01n02 is fenced
  550. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Fence (reboot) el8-a01n01 'drbd:1 failed there'
  551. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Fence (reboot) el8-a01n02 'drbd:0 failed there'
  552. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:0 ( el8-a01n02 ) due to node availability
  553. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:1 ( el8-a01n01 ) due to node availability
  554. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Calculated transition 3 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-2.bz2
  555. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n02 at Jul 14 05:11:23 2020
  556. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:0 on el8-a01n02 at Jul 14 05:11:23 2020
  557. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Cluster node el8-a01n02 will be fenced: drbd:0 failed there
  558. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n01 at Jul 14 05:11:23 2020
  559. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Unexpected result (OCF_TIMEOUT) was recorded for stop of drbd:1 on el8-a01n01 at Jul 14 05:11:23 2020
  560. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Cluster node el8-a01n01 will be fenced: drbd:1 failed there
  561. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  562. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n01 after 1000000 failures (max=1000000)
  563. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  564. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Forcing drbd-clone away from el8-a01n02 after 1000000 failures (max=1000000)
  565. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Scheduling Node el8-a01n01 for STONITH
  566. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Stop of failed resource drbd:1 is implicit after el8-a01n01 is fenced
  567. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Scheduling Node el8-a01n02 for STONITH
  568. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: Stop of failed resource drbd:0 is implicit after el8-a01n02 is fenced
  569. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Fence (reboot) el8-a01n01 'drbd:1 failed there'
  570. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Fence (reboot) el8-a01n02 'drbd:0 failed there'
  571. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:0 ( el8-a01n02 ) due to node availability
  572. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: notice: * Stop drbd:1 ( el8-a01n01 ) due to node availability
  573. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-schedulerd[2587]: warning: Calculated transition 4 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-3.bz2
  574. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Requesting fencing (reboot) of node el8-a01n01
  575. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Client pacemaker-controld.2588.ef14dd15 wants to fence (reboot) 'el8-a01n01' with device '(any)'
  576. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Requesting peer fencing (reboot) targeting el8-a01n01
  577. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node1 is eligible to fence (reboot) el8-a01n01: static-list
  578. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node2 is not eligible to fence (reboot) el8-a01n01: static-list
  579. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Requesting that el8-a01n02 perform 'reboot' action targeting el8-a01n01
  580. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node1 is eligible to fence (reboot) el8-a01n01: static-list
  581. Jul 14 05:13:03 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node2 is not eligible to fence (reboot) el8-a01n01: static-list
  582. Jul 14 05:13:20 el8-a01n02.digimer.ca corosync[2570]: [KNET ] link: host: 1 link: 0 is down
  583. Jul 14 05:13:20 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
  584. Jul 14 05:13:20 el8-a01n02.digimer.ca corosync[2570]: [KNET ] host: host: 1 has no active links
  585. Jul 14 05:13:21 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] Token has not been received in 750 ms
  586. Jul 14 05:13:21 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] A processor failed, forming new configuration.
  587. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Operation 'reboot' [4736] (call 3 from pacemaker-controld.2588) for host 'el8-a01n01' with device 'virsh_node1' returned: 0 (OK)
  588. Jul 14 05:13:22 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] A new membership (2.5f) was formed. Members left: 1
  589. Jul 14 05:13:22 el8-a01n02.digimer.ca corosync[2570]: [TOTEM ] Failed to receive the leave message. failed: 1
  590. Jul 14 05:13:22 el8-a01n02.digimer.ca corosync[2570]: [CPG ] downlist left_list: 1 received
  591. Jul 14 05:13:22 el8-a01n02.digimer.ca corosync[2570]: [QUORUM] Members[1]: 2
  592. Jul 14 05:13:22 el8-a01n02.digimer.ca corosync[2570]: [MAIN ] Completed service synchronization, ready to provide service.
  593. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemakerd[2582]: notice: Node el8-a01n01 state is now lost
  594. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Node el8-a01n01 state is now lost
  595. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Node el8-a01n01 state is now lost
  596. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Purged 1 peer with id=1 and/or uname=el8-a01n01 from the membership cache
  597. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Node el8-a01n01 state is now lost
  598. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Removing all el8-a01n01 attributes for peer loss
  599. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-attrd[2586]: notice: Purged 1 peer with id=1 and/or uname=el8-a01n01 from the membership cache
  600. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Node el8-a01n01 state is now lost
  601. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-based[2583]: notice: Purged 1 peer with id=1 and/or uname=el8-a01n01 from the membership cache
  602. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Operation 'reboot' targeting el8-a01n01 on el8-a01n02 for pacemaker-controld.2588@el8-a01n02.8001dd33: OK
  603. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Stonith operation 3/4:4:0:b6f23359-f78c-4ce2-9ca9-4e2109f4d093: OK (0)
  604. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Peer el8-a01n01 was terminated (reboot) by el8-a01n02 on behalf of pacemaker-controld.2588: OK
  605. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-controld[2588]: notice: Requesting fencing (reboot) of node el8-a01n02
  606. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Client pacemaker-controld.2588.ef14dd15 wants to fence (reboot) 'el8-a01n02' with device '(any)'
  607. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Requesting peer fencing (reboot) targeting el8-a01n02
  608. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  609. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
  610. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: Requesting that el8-a01n02 perform 'reboot' action targeting el8-a01n02
  611. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node1 is not eligible to fence (reboot) el8-a01n02: static-list
  612. Jul 14 05:13:22 el8-a01n02.digimer.ca pacemaker-fenced[2584]: notice: virsh_node2 is eligible to fence (reboot) el8-a01n02: static-list
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!