API tools faq
paste
Advertisement
cmassey

pacemaker kvm log

cmassey
Oct 31st, 2011
187
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 131.90 KB | None | 0 0
raw download clone embed print report
  1. Nov 1 03:13:35 admin01 corosync[15337]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
  2. Nov 1 03:13:35 admin01 corosync[15337]: [TOTEM ] Initializing transport (UDP/IP Multicast).
  3. Nov 1 03:13:35 admin01 corosync[15337]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
  4. Nov 1 03:13:35 admin01 corosync[15337]: [TOTEM ] The network interface [2.1.1.1] is now up.
  5. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: process_ais_conf: Reading configure
  6. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_init: Local handle: 9213452461992312835 for logging
  7. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_next: Processing additional logging options...
  8. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Found 'on' for option: debug
  9. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Defaulting to 'off' for option: to_logfile
  10. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog
  11. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Found 'daemon' for option: syslog_facility
  12. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_init: Local handle: 2013064636357672964 for quorum
  13. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_next: No additional configuration supplied for: quorum
  14. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: No default for option: provider
  15. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_init: Local handle: 4730966301143465989 for service
  16. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: config_find_next: Processing additional service options...
  17. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Found '0' for option: ver
  18. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
  19. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
  20. Nov 1 03:13:35 admin01 lrmd: [15346]: info: G_main_add_SignalHandler: Added signal handler for signal 15
  21. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: Invoked: /usr/lib64/heartbeat/stonithd
  22. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root
  23. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: G_main_add_SignalHandler: Added signal handler for signal 17
  24. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: get_cluster_type: Cluster type is: 'openais'.
  25. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
  26. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
  27. Nov 1 03:13:35 admin01 cib: [15345]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
  28. Nov 1 03:13:35 admin01 cib: [15345]: info: G_main_add_TriggerHandler: Added signal manual handler
  29. Nov 1 03:13:35 admin01 cib: [15345]: info: G_main_add_SignalHandler: Added signal handler for signal 17
  30. Nov 1 03:13:35 admin01 cib: [15345]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
  31. Nov 1 03:13:35 admin01 attrd: [15347]: info: Invoked: /usr/lib64/heartbeat/attrd
  32. Nov 1 03:13:35 admin01 pengine: [15348]: info: Invoked: /usr/lib64/heartbeat/pengine
  33. Nov 1 03:13:35 admin01 attrd: [15347]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
  34. Nov 1 03:13:35 admin01 attrd: [15347]: info: main: Starting up
  35. Nov 1 03:13:35 admin01 lrmd: [15346]: info: G_main_add_SignalHandler: Added signal handler for signal 17
  36. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
  37. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: init_ais_connection_classic: AIS connection established
  38. Nov 1 03:13:35 admin01 attrd: [15347]: info: get_cluster_type: Cluster type is: 'openais'.
  39. Nov 1 03:13:35 admin01 lrmd: [15346]: info: enabling coredumps
  40. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_startup: CRM: Initialized
  41. Nov 1 03:13:35 admin01 pengine: [15348]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
  42. Nov 1 03:13:35 admin01 attrd: [15347]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
  43. Nov 1 03:13:35 admin01 lrmd: [15346]: info: G_main_add_SignalHandler: Added signal handler for signal 10
  44. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: get_ais_nodeid: Server details: id=16843010 uname=admin01 cname=pcmk
  45. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] Logging: Initialized pcmk_startup
  46. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
  47. Nov 1 03:13:35 admin01 attrd: [15347]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
  48. Nov 1 03:13:35 admin01 lrmd: [15346]: info: G_main_add_SignalHandler: Added signal handler for signal 12
  49. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
  50. Nov 1 03:13:35 admin01 lrmd: [15346]: info: Started.
  51. Nov 1 03:13:35 admin01 crmd: [15349]: info: Invoked: /usr/lib64/heartbeat/crmd
  52. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_startup: Service: 10
  53. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_startup: Local hostname: admin01
  54. Nov 1 03:13:35 admin01 attrd: [15347]: info: init_ais_connection_classic: AIS connection established
  55. Nov 1 03:13:35 admin01 crmd: [15349]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
  56. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_update_nodeid: Local node id: 16843010
  57. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: crm_new_peer: Node admin01 now has id: 16843010
  58. Nov 1 03:13:35 admin01 attrd: [15347]: info: get_ais_nodeid: Server details: id=16843010 uname=admin01 cname=pcmk
  59. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: crm_new_peer: Node 16843010 is now known as admin01
  60. Nov 1 03:13:35 admin01 pengine: [15348]: info: main: Starting pengine
  61. Nov 1 03:13:35 admin01 stonith-ng: [15344]: info: main: Starting stonith-ng mainloop
  62. Nov 1 03:13:35 admin01 crmd: [15349]: info: main: CRM Hg Version: 01e86afaaa6d4a8c4836f68df80ababd6ca3902f
  63. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: update_member: Creating entry for node 16843010 born on 0
  64. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: update_member: 0xe096f0 Node 16843010 now known as admin01 (was: (null))
  65. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: update_member: Node admin01 now has 1 quorum votes (was 0)
  66. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: update_member: Node 16843010/admin01 is now: member
  67. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15344 for process stonith-ng
  68. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15345 for process cib
  69. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15346 for process lrmd
  70. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15347 for process attrd
  71. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15348 for process pengine
  72. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: spawn_child: Forked child 15349 for process crmd
  73. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.5
  74. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync extended virtual synchrony service
  75. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync configuration service
  76. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
  77. Nov 1 03:13:35 admin01 attrd: [15347]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
  78. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync cluster config database access v1.01
  79. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync profile loading service
  80. Nov 1 03:13:35 admin01 attrd: [15347]: info: crm_new_peer: Node admin01 now has id: 16843010
  81. Nov 1 03:13:35 admin01 attrd: [15347]: info: crm_new_peer: Node 16843010 is now known as admin01
  82. Nov 1 03:13:35 admin01 attrd: [15347]: info: main: Cluster connection active
  83. Nov 1 03:13:35 admin01 corosync[15337]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
  84. Nov 1 03:13:35 admin01 crmd: [15349]: info: crmd_init: Starting crmd
  85. Nov 1 03:13:35 admin01 corosync[15337]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
  86. Nov 1 03:13:35 admin01 attrd: [15347]: info: main: Accepting attribute updates
  87. Nov 1 03:13:35 admin01 attrd: [15347]: info: main: Starting mainloop...
  88. Nov 1 03:13:35 admin01 crmd: [15349]: info: G_main_add_SignalHandler: Added signal handler for signal 17
  89. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 316: memb=0, new=0, lost=0
  90. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 316: memb=1, new=1, lost=0
  91. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: NEW: admin01 16843010
  92. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: MEMB: admin01 16843010
  93. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: update_member: Node admin01 now has process list: 00000000000000000000000000111312 (1118994)
  94. Nov 1 03:13:35 admin01 corosync[15337]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  95. Nov 1 03:13:35 admin01 corosync[15337]: [CPG ] downlist received left_list: 0
  96. Nov 1 03:13:35 admin01 corosync[15337]: [CPG ] chosen downlist from node r(0) ip(2.1.1.1)
  97. Nov 1 03:13:35 admin01 corosync[15337]: [MAIN ] Completed service synchronization, ready to provide service.
  98. Nov 1 03:13:35 admin01 cib: [15345]: info: validate_with_relaxng: Creating RNG parser context
  99. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Recorded connection 0xe143a0 for stonith-ng/15344
  100. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Recorded connection 0xe217e0 for attrd/15347
  101. Nov 1 03:13:35 admin01 cib: [15345]: info: startCib: CIB Initialization completed successfully
  102. Nov 1 03:13:35 admin01 cib: [15345]: info: get_cluster_type: Cluster type is: 'openais'.
  103. Nov 1 03:13:35 admin01 cib: [15345]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
  104. Nov 1 03:13:35 admin01 cib: [15345]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
  105. Nov 1 03:13:35 admin01 cib: [15345]: info: init_ais_connection_classic: AIS connection established
  106. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Recorded connection 0xe1ccd0 for cib/15345
  107. Nov 1 03:13:35 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Sending membership update 316 to cib
  108. Nov 1 03:13:35 admin01 cib: [15345]: info: get_ais_nodeid: Server details: id=16843010 uname=admin01 cname=pcmk
  109. Nov 1 03:13:35 admin01 cib: [15345]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
  110. Nov 1 03:13:35 admin01 cib: [15345]: info: crm_new_peer: Node admin01 now has id: 16843010
  111. Nov 1 03:13:35 admin01 cib: [15345]: info: crm_new_peer: Node 16843010 is now known as admin01
  112. Nov 1 03:13:35 admin01 cib: [15345]: info: cib_init: Starting cib mainloop
  113. Nov 1 03:13:35 admin01 cib: [15345]: info: ais_dispatch_message: Membership 316: quorum still lost
  114. Nov 1 03:13:35 admin01 cib: [15345]: info: crm_update_peer: Node admin01: id=16843010 state=member (new) addr=r(0) ip(2.1.1.1) (new) votes=1 (new) born=0 seen=316 proc=00000000000000000000000000111312 (new)
  115. Nov 1 03:13:35 admin01 cib: [15345]: info: Managed write_cib_contents process 15353 exited with return code 0.
  116. Nov 1 03:13:36 admin01 crmd: [15349]: info: do_cib_control: CIB connection established
  117. Nov 1 03:13:36 admin01 crmd: [15349]: info: get_cluster_type: Cluster type is: 'openais'.
  118. Nov 1 03:13:36 admin01 crmd: [15349]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
  119. Nov 1 03:13:36 admin01 crmd: [15349]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
  120. Nov 1 03:13:36 admin01 crmd: [15349]: info: init_ais_connection_classic: AIS connection established
  121. Nov 1 03:13:36 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Recorded connection 0xe25b40 for crmd/15349
  122. Nov 1 03:13:36 admin01 corosync[15337]: [pcmk ] info: pcmk_ipc: Sending membership update 316 to crmd
  123. Nov 1 03:13:36 admin01 crmd: [15349]: info: get_ais_nodeid: Server details: id=16843010 uname=admin01 cname=pcmk
  124. Nov 1 03:13:36 admin01 crmd: [15349]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
  125. Nov 1 03:13:36 admin01 crmd: [15349]: info: crm_new_peer: Node admin01 now has id: 16843010
  126. Nov 1 03:13:36 admin01 crmd: [15349]: info: crm_new_peer: Node 16843010 is now known as admin01
  127. Nov 1 03:13:36 admin01 crmd: [15349]: info: ais_status_callback: status: admin01 is now unknown
  128. Nov 1 03:13:36 admin01 crmd: [15349]: info: do_ha_control: Connected to the cluster
  129. Nov 1 03:13:36 admin01 crmd: [15349]: info: do_started: Delaying start, no membership data (0000000000100000)
  130. Nov 1 03:13:36 admin01 crmd: [15349]: info: crmd_init: Starting crmd's mainloop
  131. Nov 1 03:13:36 admin01 crmd: [15349]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
  132. Nov 1 03:13:36 admin01 crmd: [15349]: info: config_query_callback: Checking for expired actions every 900000ms
  133. Nov 1 03:13:36 admin01 crmd: [15349]: info: config_query_callback: Sending expected-votes=2 to corosync
  134. Nov 1 03:13:36 admin01 crmd: [15349]: info: ais_dispatch_message: Membership 316: quorum still lost
  135. Nov 1 03:13:36 admin01 crmd: [15349]: notice: crmd_peer_update: Status update: Client admin01/crmd now has status [online] (DC=<null>)
  136. Nov 1 03:13:36 admin01 crmd: [15349]: info: ais_status_callback: status: admin01 is now member (was unknown)
  137. Nov 1 03:13:36 admin01 crmd: [15349]: info: crm_update_peer: Node admin01: id=16843010 state=member (new) addr=r(0) ip(2.1.1.1) (new) votes=1 (new) born=0 seen=316 proc=00000000000000000000000000111312 (new)
  138. Nov 1 03:13:36 admin01 crmd: [15349]: info: do_started: The local CRM is operational
  139. Nov 1 03:13:36 admin01 crmd: [15349]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
  140. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 320: memb=1, new=0, lost=0
  141. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: memb: admin01 16843010
  142. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 320: memb=2, new=1, lost=0
  143. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: Creating entry for node 33620226 born on 320
  144. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: Node 33620226/unknown is now: member
  145. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 33620226
  146. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: MEMB: admin01 16843010
  147. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 33620226
  148. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: send_member_notification: Sending membership update 320 to 2 children
  149. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: 0xe096f0 Node 16843010 ((null)) born on: 320
  150. Nov 1 03:13:37 admin01 cib: [15345]: info: ais_dispatch_message: Membership 320: quorum still lost
  151. Nov 1 03:13:37 admin01 corosync[15337]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
  152. Nov 1 03:13:37 admin01 cib: [15345]: info: crm_new_peer: Node <null> now has id: 33620226
  153. Nov 1 03:13:37 admin01 cib: [15345]: info: crm_update_peer: Node (null): id=33620226 state=member (new) addr=r(0) ip(2.1.1.2) votes=0 born=0 seen=320 proc=00000000000000000000000000000000
  154. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: 0xe19310 Node 33620226 (admin02) born on: 320
  155. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: 0xe19310 Node 33620226 now known as admin02 (was: (null))
  156. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: Node admin02 now has process list: 00000000000000000000000000111312 (1118994)
  157. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: update_member: Node admin02 now has 1 quorum votes (was 0)
  158. Nov 1 03:13:37 admin01 corosync[15337]: [pcmk ] info: send_member_notification: Sending membership update 320 to 2 children
  159. Nov 1 03:13:37 admin01 cib: [15345]: notice: ais_dispatch_message: Membership 320: quorum acquired
  160. Nov 1 03:13:37 admin01 cib: [15345]: info: crm_get_peer: Node 33620226 is now known as admin02
  161. Nov 1 03:13:37 admin01 cib: [15345]: info: crm_update_peer: Node admin02: id=33620226 state=member addr=r(0) ip(2.1.1.2) votes=1 (new) born=320 seen=320 proc=00000000000000000000000000111312 (new)
  162. Nov 1 03:13:37 admin01 corosync[15337]: [CPG ] downlist received left_list: 0
  163. Nov 1 03:13:37 admin01 corosync[15337]: [CPG ] downlist received left_list: 0
  164. Nov 1 03:13:37 admin01 corosync[15337]: [CPG ] chosen downlist from node r(0) ip(2.1.1.1)
  165. Nov 1 03:13:37 admin01 corosync[15337]: [MAIN ] Completed service synchronization, ready to provide service.
  166. Nov 1 03:13:37 admin01 crmd: [15349]: info: ais_dispatch_message: Membership 316: quorum still lost
  167. Nov 1 03:13:37 admin01 crmd: [15349]: info: ais_dispatch_message: Membership 320: quorum still lost
  168. Nov 1 03:13:37 admin01 crmd: [15349]: info: crm_new_peer: Node <null> now has id: 33620226
  169. Nov 1 03:13:37 admin01 crmd: [15349]: info: crm_update_peer: Node (null): id=33620226 state=member (new) addr=r(0) ip(2.1.1.2) votes=0 born=0 seen=320 proc=00000000000000000000000000000000
  170. Nov 1 03:13:37 admin01 crmd: [15349]: notice: ais_dispatch_message: Membership 320: quorum acquired
  171. Nov 1 03:13:37 admin01 crmd: [15349]: info: crm_get_peer: Node 33620226 is now known as admin02
  172. Nov 1 03:13:37 admin01 crmd: [15349]: info: ais_status_callback: status: admin02 is now member
  173. Nov 1 03:13:37 admin01 crmd: [15349]: notice: crmd_peer_update: Status update: Client admin02/crmd now has status [online] (DC=<null>)
  174. Nov 1 03:13:37 admin01 crmd: [15349]: info: crm_update_peer: Node admin02: id=33620226 state=member addr=r(0) ip(2.1.1.2) votes=1 (new) born=320 seen=320 proc=00000000000000000000000000111312 (new)
  175. Nov 1 03:13:37 admin01 crmd: [15349]: info: te_connect_stonith: Attempting connection to fencing daemon...
  176. Nov 1 03:13:38 admin01 crmd: [15349]: info: te_connect_stonith: Connected
  177. Nov 1 03:13:40 admin01 attrd: [15347]: info: cib_connect: Connected to the CIB after 1 signon attempts
  178. Nov 1 03:13:40 admin01 attrd: [15347]: info: cib_connect: Sending full refresh
  179. Nov 1 03:14:37 admin01 crmd: [15349]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! (60000ms)
  180. Nov 1 03:14:37 admin01 crmd: [15349]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
  181. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
  182. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
  183. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_te_control: Registering TE UUID: c5a13ba5-3b1f-4b81-82a2-838830e194a8
  184. Nov 1 03:14:37 admin01 crmd: [15349]: WARN: cib_client_add_notify_callback: Callback already present
  185. Nov 1 03:14:37 admin01 crmd: [15349]: info: set_graph_functions: Setting custom graph functions
  186. Nov 1 03:14:37 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
  187. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_takeover: Taking over DC status for this partition
  188. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_readwrite: We are now in R/W mode
  189. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/5, version=0.27.1): ok (rc=0)
  190. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/6, version=0.27.2): ok (rc=0)
  191. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/8, version=0.27.3): ok (rc=0)
  192. Nov 1 03:14:37 admin01 crmd: [15349]: info: join_make_offer: Making join offers based on membership 320
  193. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_join_offer_all: join-1: Waiting on 2 outstanding join acks
  194. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/10, version=0.27.4): ok (rc=0)
  195. Nov 1 03:14:37 admin01 crmd: [15349]: info: ais_dispatch_message: Membership 320: quorum retained
  196. Nov 1 03:14:37 admin01 crmd: [15349]: info: crmd_ais_dispatch: Setting expected votes to 2
  197. Nov 1 03:14:37 admin01 crmd: [15349]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
  198. Nov 1 03:14:37 admin01 crmd: [15349]: info: config_query_callback: Checking for expired actions every 900000ms
  199. Nov 1 03:14:37 admin01 crmd: [15349]: info: config_query_callback: Sending expected-votes=2 to corosync
  200. Nov 1 03:14:37 admin01 crmd: [15349]: info: update_dc: Set DC to admin01 (3.0.5)
  201. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/13, version=0.27.5): ok (rc=0)
  202. Nov 1 03:14:37 admin01 crmd: [15349]: info: ais_dispatch_message: Membership 320: quorum retained
  203. Nov 1 03:14:37 admin01 crmd: [15349]: info: crmd_ais_dispatch: Setting expected votes to 2
  204. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/16, version=0.27.6): ok (rc=0)
  205. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
  206. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
  207. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_join_finalize: join-1: Syncing the CIB from admin01 to the rest of the cluster
  208. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/17, version=0.27.6): ok (rc=0)
  209. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/18, version=0.27.7): ok (rc=0)
  210. Nov 1 03:14:37 admin01 crmd: [15349]: info: update_attrd: Connecting to attrd...
  211. Nov 1 03:14:37 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for terminate
  212. Nov 1 03:14:37 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for shutdown
  213. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_join_ack: join-1: Updating node state to member for admin02
  214. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_join_ack: join-1: Updating node state to member for admin01
  215. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/19, version=0.27.8): ok (rc=0)
  216. Nov 1 03:14:37 admin01 attrd: [15347]: info: crm_new_peer: Node admin02 now has id: 33620226
  217. Nov 1 03:14:37 admin01 attrd: [15347]: info: crm_new_peer: Node 33620226 is now known as admin02
  218. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='admin01']/transient_attributes (origin=local/crmd/20, version=0.27.9): ok (rc=0)
  219. Nov 1 03:14:37 admin01 crmd: [15349]: info: erase_xpath_callback: Deletion of "//node_state[@uname='admin01']/transient_attributes": ok (rc=0)
  220. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='admin02']/lrm (origin=local/crmd/21, version=0.27.10): ok (rc=0)
  221. Nov 1 03:14:37 admin01 crmd: [15349]: info: erase_xpath_callback: Deletion of "//node_state[@uname='admin02']/lrm": ok (rc=0)
  222. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
  223. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  224. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
  225. Nov 1 03:14:37 admin01 crmd: [15349]: info: crm_update_quorum: Updating quorum status to true (call=27)
  226. Nov 1 03:14:37 admin01 crmd: [15349]: info: abort_transition_graph: do_te_invoke:173 - Triggered transition abort (complete=1) : Peer Cancelled
  227. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_pe_invoke: Query 28: Requesting the current CIB: S_POLICY_ENGINE
  228. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='admin02']/transient_attributes (origin=admin02/crmd/7, version=0.27.12): ok (rc=0)
  229. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='admin01']/lrm (origin=local/crmd/23, version=0.27.13): ok (rc=0)
  230. Nov 1 03:14:37 admin01 crmd: [15349]: info: erase_xpath_callback: Deletion of "//node_state[@uname='admin01']/lrm": ok (rc=0)
  231. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/25, version=0.27.15): ok (rc=0)
  232. Nov 1 03:14:37 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/27, version=0.27.17): ok (rc=0)
  233. Nov 1 03:14:37 admin01 attrd: [15347]: info: attrd_local_callback: Sending full refresh (origin=crmd)
  234. Nov 1 03:14:37 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
  235. Nov 1 03:14:37 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
  236. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=28, ref=pe_calc-dc-1320117277-11, seq=320, quorate=1
  237. Nov 1 03:14:37 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  238. Nov 1 03:14:37 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  239. Nov 1 03:14:37 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  240. Nov 1 03:14:37 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  241. Nov 1 03:14:37 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  242. Nov 1 03:14:37 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Stopped
  243. Nov 1 03:14:37 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  244. Nov 1 03:14:37 admin01 pengine: [15348]: notice: short_print: Stopped: [ convirt-drbd:0 convirt-drbd:1 ]
  245. Nov 1 03:14:37 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Stopped
  246. Nov 1 03:14:37 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  247. Nov 1 03:14:37 admin01 pengine: [15348]: notice: short_print: Stopped: [ sitescope-drbd:0 sitescope-drbd:1 ]
  248. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:0 (Stopped admin01)
  249. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  250. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:0 (Stopped admin01)
  251. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  252. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 0 instances of a possible 1 to master
  253. Nov 1 03:14:37 admin01 pengine: [15348]: info: native_color: Resource sitescope-kvm cannot run anywhere
  254. Nov 1 03:14:37 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 0 instances of a possible 1 to master
  255. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-kvm on admin01
  256. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:0 on admin01
  257. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  258. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  259. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:0 on admin01
  260. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  261. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  262. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:0 on admin02
  263. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:0 on admin02
  264. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  265. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  266. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:0 on admin02
  267. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:0 on admin02
  268. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  269. Nov 1 03:14:37 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  270. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Start convirt-kvm#011(admin01)
  271. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Start convirt-drbd:0#011(admin01)
  272. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Promote convirt-drbd:0#011(Stopped -> Master admin01)
  273. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Start convirt-drbd:1#011(admin02)
  274. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-kvm#011(Stopped)
  275. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Start sitescope-drbd:0#011(admin02)
  276. Nov 1 03:14:37 admin01 pengine: [15348]: notice: LogActions: Start sitescope-drbd:1#011(admin01)
  277. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  278. Nov 1 03:14:37 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 0: 51 actions in 51 synapses
  279. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1320117277-11) derived from /var/lib/pengine/pe-input-2698.bz2
  280. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 9: monitor convirt-kvm_monitor_0 on admin02
  281. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 4: monitor convirt-kvm_monitor_0 on admin01 (local)
  282. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=4:0:7:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-kvm_monitor_0 )
  283. Nov 1 03:14:37 admin01 lrmd: [15346]: info: rsc:convirt-kvm:2: probe
  284. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 5: monitor convirt-drbd:0_monitor_0 on admin01 (local)
  285. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=5:0:7:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_monitor_0 )
  286. Nov 1 03:14:37 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:3: probe
  287. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 10: monitor convirt-drbd:1_monitor_0 on admin02
  288. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
  289. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 11: monitor sitescope-kvm_monitor_0 on admin02
  290. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 6: monitor sitescope-kvm_monitor_0 on admin01 (local)
  291. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=6:0:7:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-kvm_monitor_0 )
  292. Nov 1 03:14:37 admin01 lrmd: [15346]: info: rsc:sitescope-kvm:4: probe
  293. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 12: monitor sitescope-drbd:0_monitor_0 on admin02
  294. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 7: monitor sitescope-drbd:1_monitor_0 on admin01 (local)
  295. Nov 1 03:14:37 admin01 pengine: [15348]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/pengine/pe-input-2698.bz2
  296. Nov 1 03:14:37 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=7:0:7:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_monitor_0 )
  297. Nov 1 03:14:37 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:5: probe
  298. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 53 fired and confirmed
  299. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
  300. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
  301. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 54 fired and confirmed
  302. Nov 1 03:14:37 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 51 fired and confirmed
  303. Nov 1 03:14:38 admin01 crm_attribute: [15454]: info: determine_host: Mapped admin01 to admin01
  304. Nov 1 03:14:38 admin01 crm_attribute: [15454]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  305. Nov 1 03:14:38 admin01 crm_attribute: [15454]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  306. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for master-convirt-drbd:0
  307. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: master-convirt-drbd:0 (5)
  308. Nov 1 03:14:38 admin01 crm_attribute: [15454]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  309. Nov 1 03:14:38 admin01 VirtualDomain[15370]: ERROR: /var/run/heartbeat/rsctmp/VirtualDomain-convirt-kvm.state is empty. This is unexpected. Cannot determine domain name.
  310. Nov 1 03:14:38 admin01 crm_attribute: [15456]: info: determine_host: Mapped admin01 to admin01
  311. Nov 1 03:14:38 admin01 crm_attribute: [15456]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  312. Nov 1 03:14:38 admin01 crm_attribute: [15456]: info: main: Update master-sitescope-drbd:1=5 sent via attrd
  313. Nov 1 03:14:38 admin01 crm_attribute: [15456]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  314. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_perform_update: Sent update 10: master-convirt-drbd:0=5
  315. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for master-sitescope-drbd:1
  316. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: master-sitescope-drbd:1 (5)
  317. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin01-master-convirt-drbd:0, magic=NA, cib=0.27.18) : Transient attribute: update
  318. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
  319. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: Abort action done superceeded by restart
  320. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_perform_update: Sent update 13: master-sitescope-drbd:1=5
  321. Nov 1 03:14:38 admin01 lrmd: [15346]: WARN: Managed convirt-kvm:monitor process 15370 exited with return code 1.
  322. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin01-master-sitescope-drbd:1, magic=NA, cib=0.27.19) : Transient attribute: update
  323. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-kvm_monitor_0 (call=2, rc=1, cib-update=29, confirmed=true) unknown error
  324. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for master-sitescope-drbd:0
  325. Nov 1 03:14:38 admin01 crmd: [15349]: WARN: status_from_rc: Action 4 (convirt-kvm_monitor_0) on admin01 failed (target: 7 vs. rc: 1): Error
  326. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: match_graph_event:265 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=convirt-kvm_monitor_0, magic=0:1;4:0:7:c5a13ba5-3b1f-4b81-82a2-838830e194a8, cib=0.27.20) : Event failed
  327. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-kvm_monitor_0 (4) confirmed on admin01 (rc=4)
  328. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin02-master-sitescope-drbd:0, magic=NA, cib=0.27.21) : Transient attribute: update
  329. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for master-convirt-drbd:1
  330. Nov 1 03:14:38 admin01 lrmd: [15346]: WARN: Managed convirt-drbd:0:monitor process 15371 exited with return code 7.
  331. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_monitor_0 (call=3, rc=7, cib-update=30, confirmed=true) not running
  332. Nov 1 03:14:38 admin01 lrmd: [15346]: WARN: Managed sitescope-drbd:1:monitor process 15373 exited with return code 7.
  333. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin02-master-convirt-drbd:1, magic=NA, cib=0.27.22) : Transient attribute: update
  334. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_monitor_0 (call=5, rc=7, cib-update=31, confirmed=true) not running
  335. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_monitor_0 (5) confirmed on admin01 (rc=0)
  336. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_monitor_0 (7) confirmed on admin01 (rc=0)
  337. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_monitor_0 (12) confirmed on admin02 (rc=0)
  338. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_monitor_0 (10) confirmed on admin02 (rc=0)
  339. Nov 1 03:14:38 admin01 lrmd: [15346]: WARN: Managed sitescope-kvm:monitor process 15372 exited with return code 7.
  340. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-kvm_monitor_0 (call=4, rc=7, cib-update=32, confirmed=true) not running
  341. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-kvm_monitor_0 (6) confirmed on admin01 (rc=0)
  342. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on admin01 (local) - no waiting
  343. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for probe_complete
  344. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
  345. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_perform_update: Sent update 18: probe_complete=true
  346. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-kvm_monitor_0 (9) confirmed on admin02 (rc=0)
  347. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-kvm_monitor_0 (11) confirmed on admin02 (rc=0)
  348. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on admin02 - no waiting
  349. Nov 1 03:14:38 admin01 crmd: [15349]: info: run_graph: ====================================================
  350. Nov 1 03:14:38 admin01 crmd: [15349]: notice: run_graph: Transition 0 (Complete=16, Pending=0, Fired=0, Skipped=20, Incomplete=15, Source=/var/lib/pengine/pe-input-2698.bz2): Stopped
  351. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_graph_trigger: Transition 0 is now complete
  352. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  353. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  354. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_pe_invoke: Query 33: Requesting the current CIB: S_POLICY_ENGINE
  355. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=33, ref=pe_calc-dc-1320117278-22, seq=320, quorate=1
  356. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  357. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  358. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  359. Nov 1 03:14:38 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  360. Nov 1 03:14:38 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  361. Nov 1 03:14:38 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  362. Nov 1 03:14:38 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 FAILED
  363. Nov 1 03:14:38 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  364. Nov 1 03:14:38 admin01 pengine: [15348]: notice: short_print: Stopped: [ convirt-drbd:0 convirt-drbd:1 ]
  365. Nov 1 03:14:38 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Stopped
  366. Nov 1 03:14:38 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  367. Nov 1 03:14:38 admin01 pengine: [15348]: notice: short_print: Stopped: [ sitescope-drbd:0 sitescope-drbd:1 ]
  368. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:0 (Stopped admin01)
  369. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  370. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:0 (Stopped admin01)
  371. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  372. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 0 instances of a possible 1 to master
  373. Nov 1 03:14:38 admin01 pengine: [15348]: info: native_color: Resource sitescope-kvm cannot run anywhere
  374. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 0 instances of a possible 1 to master
  375. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-kvm on admin01
  376. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:0 on admin01
  377. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  378. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  379. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:0 on admin01
  380. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  381. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  382. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:0 on admin02
  383. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:0 on admin02
  384. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  385. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  386. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:0 on admin02
  387. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:0 on admin02
  388. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  389. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  390. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Recover convirt-kvm#011(Started admin01)
  391. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Start convirt-drbd:0#011(admin01)
  392. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Promote convirt-drbd:0#011(Stopped -> Master admin01)
  393. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Start convirt-drbd:1#011(admin02)
  394. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-kvm#011(Stopped)
  395. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Start sitescope-drbd:0#011(admin02)
  396. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Start sitescope-drbd:1#011(admin01)
  397. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  398. Nov 1 03:14:38 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 1: 43 actions in 43 synapses
  399. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1320117278-22) derived from /var/lib/pengine/pe-input-2699.bz2
  400. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 1: stop convirt-kvm_stop_0 on admin01 (local)
  401. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=1:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-kvm_stop_0 )
  402. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:convirt-kvm:6: stop
  403. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
  404. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 46 fired and confirmed
  405. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on admin02 - no waiting
  406. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
  407. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 14 fired and confirmed
  408. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
  409. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 44 fired and confirmed
  410. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 8: start convirt-drbd:0_start_0 on admin01 (local)
  411. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=8:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_start_0 )
  412. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:7: start
  413. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 11: start convirt-drbd:1_start_0 on admin02
  414. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 38: start sitescope-drbd:0_start_0 on admin02
  415. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 41: start sitescope-drbd:1_start_0 on admin01 (local)
  416. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=41:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_start_0 )
  417. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:8: start
  418. Nov 1 03:14:38 admin01 pengine: [15348]: info: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-2699.bz2
  419. Nov 1 03:14:38 admin01 VirtualDomain[15487]: ERROR: /var/run/heartbeat/rsctmp/VirtualDomain-convirt-kvm.state is empty. This is unexpected. Cannot determine domain name.
  420. Nov 1 03:14:38 admin01 lrmd: [15346]: WARN: Managed convirt-kvm:stop process 15487 exited with return code 1.
  421. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-kvm_stop_0 (call=6, rc=1, cib-update=34, confirmed=true) unknown error
  422. Nov 1 03:14:38 admin01 crmd: [15349]: WARN: status_from_rc: Action 1 (convirt-kvm_stop_0) on admin01 failed (target: 0 vs. rc: 1): Error
  423. Nov 1 03:14:38 admin01 crmd: [15349]: WARN: update_failcount: Updating failcount for convirt-kvm on admin01 after failed stop: rc=1 (update=INFINITY, time=1320117278)
  424. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: match_graph_event:265 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=convirt-kvm_stop_0, magic=0:1;1:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8, cib=0.27.33) : Event failed
  425. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 0 to 1
  426. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: Abort action done superceeded by restart
  427. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-kvm_stop_0 (1) confirmed on admin01 (rc=4)
  428. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for fail-count-convirt-kvm
  429. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-convirt-kvm (INFINITY)
  430. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_perform_update: Sent update 23: fail-count-convirt-kvm=INFINITY
  431. Nov 1 03:14:38 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for last-failure-convirt-kvm
  432. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-convirt-kvm (1320117278)
  433. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin01-fail-count-convirt-kvm, magic=NA, cib=0.27.34) : Transient attribute: update
  434. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 1 to 1000000
  435. Nov 1 03:14:38 admin01 crmd: [15349]: info: update_abort_priority: 'Event failed' abort superceeded
  436. Nov 1 03:14:38 admin01 attrd: [15347]: info: attrd_perform_update: Sent update 26: last-failure-convirt-kvm=1320117278
  437. Nov 1 03:14:38 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin01-last-failure-convirt-kvm, magic=NA, cib=0.27.35) : Transient attribute: update
  438. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  439. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  440. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  441. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  442. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  443. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  444. Nov 1 03:14:38 admin01 kernel: d-con convirt: Starting worker thread (from drbdsetup [15661])
  445. Nov 1 03:14:38 admin01 kernel: block drbd0: disk( Diskless -> Attaching )
  446. Nov 1 03:14:38 admin01 kernel: block drbd0: Method to ensure write ordering: barrier
  447. Nov 1 03:14:38 admin01 kernel: block drbd0: max BIO size = 524288
  448. Nov 1 03:14:38 admin01 kernel: block drbd0: drbd_bm_resize called with capacity == 46135864
  449. Nov 1 03:14:38 admin01 kernel: block drbd0: resync bitmap: bits=5766983 words=90110 pages=176
  450. Nov 1 03:14:38 admin01 kernel: block drbd0: size = 22 GB (23067932 KB)
  451. Nov 1 03:14:38 admin01 kernel: block drbd0: bitmap READ of 176 pages took 31 jiffies
  452. Nov 1 03:14:38 admin01 kernel: block drbd0: recounting of set bits took additional 1 jiffies
  453. Nov 1 03:14:38 admin01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  454. Nov 1 03:14:38 admin01 kernel: block drbd0: disk( Attaching -> Consistent )
  455. Nov 1 03:14:38 admin01 kernel: block drbd0: attached to UUIDs B21A3767BB62B1E0:0000000000000000:C9D11CAB529FACB6:C9D01CAB529FACB6
  456. Nov 1 03:14:38 admin01 kernel: d-con sitescope: Starting worker thread (from drbdsetup [15663])
  457. Nov 1 03:14:38 admin01 kernel: block drbd1: disk( Diskless -> Attaching )
  458. Nov 1 03:14:38 admin01 kernel: block drbd1: Method to ensure write ordering: barrier
  459. Nov 1 03:14:38 admin01 kernel: block drbd1: max BIO size = 524288
  460. Nov 1 03:14:38 admin01 kernel: block drbd1: drbd_bm_resize called with capacity == 46135864
  461. Nov 1 03:14:38 admin01 kernel: block drbd1: resync bitmap: bits=5766983 words=90110 pages=176
  462. Nov 1 03:14:38 admin01 kernel: block drbd1: size = 22 GB (23067932 KB)
  463. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  464. Nov 1 03:14:38 admin01 kernel: block drbd1: bitmap READ of 176 pages took 10 jiffies
  465. Nov 1 03:14:38 admin01 kernel: block drbd1: recounting of set bits took additional 1 jiffies
  466. Nov 1 03:14:38 admin01 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  467. Nov 1 03:14:38 admin01 kernel: block drbd1: disk( Attaching -> Consistent )
  468. Nov 1 03:14:38 admin01 kernel: block drbd1: attached to UUIDs 288A56ECEB2D80B2:0000000000000000:FB626992EDD04F6A:FB616992EDD04F6A
  469. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  470. Nov 1 03:14:38 admin01 kernel: d-con convirt: conn( StandAlone -> Unconnected )
  471. Nov 1 03:14:38 admin01 kernel: d-con convirt: Starting receiver thread (from drbd_w_convirt [15662])
  472. Nov 1 03:14:38 admin01 kernel: d-con convirt: receiver (re)started
  473. Nov 1 03:14:38 admin01 kernel: d-con convirt: conn( Unconnected -> WFConnection )
  474. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  475. Nov 1 03:14:38 admin01 kernel: d-con sitescope: conn( StandAlone -> Unconnected )
  476. Nov 1 03:14:38 admin01 kernel: d-con sitescope: Starting receiver thread (from drbd_w_sitescop [15666])
  477. Nov 1 03:14:38 admin01 kernel: d-con sitescope: receiver (re)started
  478. Nov 1 03:14:38 admin01 kernel: d-con sitescope: conn( Unconnected -> WFConnection )
  479. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  480. Nov 1 03:14:38 admin01 crm_attribute: [15734]: info: determine_host: Mapped admin01 to admin01
  481. Nov 1 03:14:38 admin01 crm_attribute: [15734]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  482. Nov 1 03:14:38 admin01 crm_attribute: [15734]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  483. Nov 1 03:14:38 admin01 crm_attribute: [15734]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  484. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:start:stdout)
  485. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:start process 15488 exited with return code 0.
  486. Nov 1 03:14:38 admin01 crm_attribute: [15747]: info: determine_host: Mapped admin01 to admin01
  487. Nov 1 03:14:38 admin01 crm_attribute: [15747]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  488. Nov 1 03:14:38 admin01 crm_attribute: [15747]: info: main: Update master-sitescope-drbd:1=5 sent via attrd
  489. Nov 1 03:14:38 admin01 crm_attribute: [15747]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  490. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_start_0 (call=7, rc=0, cib-update=35, confirmed=true) ok
  491. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_start_0 (8) confirmed on admin01 (rc=0)
  492. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:start:stdout)
  493. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:start process 15489 exited with return code 0.
  494. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_start_0 (call=8, rc=0, cib-update=36, confirmed=true) ok
  495. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_start_0 (41) confirmed on admin01 (rc=0)
  496. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_start_0 (11) confirmed on admin02 (rc=0)
  497. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
  498. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
  499. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 77: notify convirt-drbd:0_post_notify_start_0 on admin01 (local)
  500. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=77:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  501. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:9: notify
  502. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 78: notify convirt-drbd:1_post_notify_start_0 on admin02
  503. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_start_0 (38) confirmed on admin02 (rc=0)
  504. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 45 fired and confirmed
  505. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
  506. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 83: notify sitescope-drbd:0_post_notify_start_0 on admin02
  507. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 84: notify sitescope-drbd:1_post_notify_start_0 on admin01 (local)
  508. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=84:1:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_notify_0 )
  509. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:10: notify
  510. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  511. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 15765 exited with return code 0.
  512. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=9, rc=0, cib-update=37, confirmed=true) ok
  513. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_start_0 (77) confirmed on admin01 (rc=0)
  514. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_post_notify_start_0 (78) confirmed on admin02 (rc=0)
  515. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
  516. Nov 1 03:14:38 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:notify:stdout)
  517. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:notify process 15768 exited with return code 0.
  518. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_notify_0 (call=10, rc=0, cib-update=38, confirmed=true) ok
  519. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_post_notify_start_0 (84) confirmed on admin01 (rc=0)
  520. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_post_notify_start_0 (83) confirmed on admin02 (rc=0)
  521. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 49 fired and confirmed
  522. Nov 1 03:14:38 admin01 crmd: [15349]: info: run_graph: ====================================================
  523. Nov 1 03:14:38 admin01 crmd: [15349]: notice: run_graph: Transition 1 (Complete=22, Pending=0, Fired=0, Skipped=16, Incomplete=5, Source=/var/lib/pengine/pe-input-2699.bz2): Stopped
  524. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_graph_trigger: Transition 1 is now complete
  525. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  526. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  527. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_pe_invoke: Query 39: Requesting the current CIB: S_POLICY_ENGINE
  528. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=39, ref=pe_calc-dc-1320117278-33, seq=320, quorate=1
  529. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  530. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  531. Nov 1 03:14:38 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  532. Nov 1 03:14:38 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  533. Nov 1 03:14:38 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  534. Nov 1 03:14:38 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  535. Nov 1 03:14:38 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_stop_0 on admin01: unknown error (1)
  536. Nov 1 03:14:38 admin01 pengine: [15348]: info: native_add_running: resource convirt-kvm isnt managed
  537. Nov 1 03:14:38 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 (unmanaged) FAILED
  538. Nov 1 03:14:38 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  539. Nov 1 03:14:38 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 admin02 ]
  540. Nov 1 03:14:38 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Stopped
  541. Nov 1 03:14:38 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  542. Nov 1 03:14:38 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin02 admin01 ]
  543. Nov 1 03:14:38 admin01 pengine: [15348]: info: get_failcount: convirt-kvm has failed INFINITY times on admin01
  544. Nov 1 03:14:38 admin01 pengine: [15348]: WARN: common_apply_stickiness: Forcing convirt-kvm away from admin01 after 1000000 failures (max=1000000)
  545. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Slave admin02)
  546. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  547. Nov 1 03:14:38 admin01 pengine: [15348]: info: native_color: Unmanaged resource convirt-kvm allocated to 'nowhere': failed
  548. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Slave admin02)
  549. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  550. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Slave admin02)
  551. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  552. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Slave admin02)
  553. Nov 1 03:14:38 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  554. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  555. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  556. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:1 on admin02
  557. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  558. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  559. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:1 on admin02
  560. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-kvm on admin02
  561. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-drbd:0 on admin02
  562. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  563. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  564. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-drbd:0 on admin02
  565. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  566. Nov 1 03:14:38 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  567. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Leave convirt-kvm#011(Started unmanaged)
  568. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:0#011(Slave admin01)
  569. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Promote convirt-drbd:1#011(Slave -> Master admin02)
  570. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Start sitescope-kvm#011(admin02)
  571. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Promote sitescope-drbd:0#011(Slave -> Master admin02)
  572. Nov 1 03:14:38 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:1#011(Slave admin01)
  573. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  574. Nov 1 03:14:38 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 2: 30 actions in 30 synapses
  575. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1320117278-33) derived from /var/lib/pengine/pe-input-2700.bz2
  576. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
  577. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 68 fired and confirmed
  578. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 90: notify convirt-drbd:0_pre_notify_promote_0 on admin01 (local)
  579. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=90:2:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  580. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:11: notify
  581. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 92: notify convirt-drbd:1_pre_notify_promote_0 on admin02
  582. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 106: notify sitescope-drbd:0_pre_notify_promote_0 on admin02
  583. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 108: notify sitescope-drbd:1_pre_notify_promote_0 on admin01 (local)
  584. Nov 1 03:14:38 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=108:2:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_notify_0 )
  585. Nov 1 03:14:38 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:12: notify
  586. Nov 1 03:14:38 admin01 pengine: [15348]: info: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-2700.bz2
  587. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 15845 exited with return code 0.
  588. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=11, rc=0, cib-update=40, confirmed=true) ok
  589. Nov 1 03:14:38 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:notify process 15846 exited with return code 0.
  590. Nov 1 03:14:38 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_notify_0 (call=12, rc=0, cib-update=41, confirmed=true) ok
  591. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_pre_notify_promote_0 (90) confirmed on admin01 (rc=0)
  592. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_pre_notify_promote_0 (108) confirmed on admin01 (rc=0)
  593. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_pre_notify_promote_0 (92) confirmed on admin02 (rc=0)
  594. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
  595. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
  596. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 16: promote convirt-drbd:1_promote_0 on admin02
  597. Nov 1 03:14:38 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_pre_notify_promote_0 (106) confirmed on admin02 (rc=0)
  598. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 69 fired and confirmed
  599. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 66 fired and confirmed
  600. Nov 1 03:14:38 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 47: promote sitescope-drbd:0_promote_0 on admin02
  601. Nov 1 03:14:39 admin01 kernel: d-con convirt: Handshake successful: Agreed network protocol version 100
  602. Nov 1 03:14:39 admin01 kernel: d-con convirt: conn( WFConnection -> WFReportParams )
  603. Nov 1 03:14:39 admin01 kernel: d-con convirt: Starting asender thread (from drbd_r_convirt [15699])
  604. Nov 1 03:14:39 admin01 kernel: block drbd0: drbd_sync_handshake:
  605. Nov 1 03:14:39 admin01 kernel: block drbd0: self B21A3767BB62B1E0:0000000000000000:C9D11CAB529FACB6:C9D01CAB529FACB6 bits:0 flags:0
  606. Nov 1 03:14:39 admin01 kernel: block drbd0: peer B21A3767BB62B1E0:0000000000000000:C9D11CAB529FACB6:C9D01CAB529FACB6 bits:0 flags:0
  607. Nov 1 03:14:39 admin01 kernel: block drbd0: uuid_compare()=0 by rule 40
  608. Nov 1 03:14:39 admin01 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> Connected ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> UpToDate )
  609. Nov 1 03:14:39 admin01 kernel: d-con sitescope: Handshake successful: Agreed network protocol version 100
  610. Nov 1 03:14:39 admin01 kernel: d-con sitescope: conn( WFConnection -> WFReportParams )
  611. Nov 1 03:14:39 admin01 kernel: d-con sitescope: Starting asender thread (from drbd_r_sitescop [15719])
  612. Nov 1 03:14:39 admin01 kernel: block drbd1: drbd_sync_handshake:
  613. Nov 1 03:14:39 admin01 kernel: block drbd1: self 288A56ECEB2D80B2:0000000000000000:FB626992EDD04F6A:FB616992EDD04F6A bits:0 flags:0
  614. Nov 1 03:14:39 admin01 kernel: block drbd1: peer 288A56ECEB2D80B2:0000000000000000:FB626992EDD04F6A:FB616992EDD04F6A bits:0 flags:0
  615. Nov 1 03:14:39 admin01 kernel: block drbd1: uuid_compare()=0 by rule 40
  616. Nov 1 03:14:39 admin01 kernel: block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> Connected ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> UpToDate )
  617. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff- <cib admin_epoch="0" epoch="27" num_updates="47" />
  618. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <cib epoch="28" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" have-quorum="1" cib-last-written="Tue Nov 1 02:43:42 2011" dc-uuid="admin01" >
  619. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <configuration >
  620. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <constraints >
  621. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <rsc_location rsc="ms-convirt" id="drbd-fence-by-handler-ms-convirt" __crm_diff_marker__="added:top" >
  622. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-rule-ms-convirt" >
  623. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ <expression attribute="#uname" operation="ne" value="admin02" id="drbd-fence-by-handler-expr-ms-convirt" />
  624. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ </rule>
  625. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ </rsc_location>
  626. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ </constraints>
  627. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ </configuration>
  628. Nov 1 03:14:39 admin01 cib: [15345]: info: cib:diff+ </cib>
  629. Nov 1 03:14:39 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_create for section constraints (origin=admin02/cibadmin/2, version=0.28.1): ok (rc=0)
  630. Nov 1 03:14:39 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:131 - Triggered transition abort (complete=0, tag=diff, id=(null), magic=NA, cib=0.28.1) : Non-status change
  631. Nov 1 03:14:39 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
  632. Nov 1 03:14:39 admin01 crmd: [15349]: info: update_abort_priority: Abort action done superceeded by restart
  633. Nov 1 03:14:39 admin01 kernel: block drbd1: peer( Secondary -> Primary )
  634. Nov 1 03:14:39 admin01 cib: [15345]: info: Managed write_cib_contents process 15891 exited with return code 0.
  635. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_promote_0 (47) confirmed on admin02 (rc=0)
  636. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 67 fired and confirmed
  637. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 70 fired and confirmed
  638. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 107: notify sitescope-drbd:0_post_notify_promote_0 on admin02
  639. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 109: notify sitescope-drbd:1_post_notify_promote_0 on admin01 (local)
  640. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=109:2:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_notify_0 )
  641. Nov 1 03:14:39 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:13: notify
  642. Nov 1 03:14:39 admin01 crmd: [15349]: WARN: status_from_rc: Action 16 (convirt-drbd:1_promote_0) on admin02 failed (target: 0 vs. rc: 1): Error
  643. Nov 1 03:14:39 admin01 crmd: [15349]: WARN: update_failcount: Updating failcount for convirt-drbd:1 on admin02 after failed promote: rc=1 (update=value++, time=1320117279)
  644. Nov 1 03:14:39 admin01 crmd: [15349]: info: abort_transition_graph: match_graph_event:265 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=convirt-drbd:1_promote_0, magic=0:1;16:2:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8, cib=0.28.3) : Event failed
  645. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_promote_0 (16) confirmed on admin02 (rc=4)
  646. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
  647. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
  648. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 91: notify convirt-drbd:0_post_notify_promote_0 on admin01 (local)
  649. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=91:2:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  650. Nov 1 03:14:39 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:14: notify
  651. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 93: notify convirt-drbd:1_post_notify_promote_0 on admin02
  652. Nov 1 03:14:39 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for fail-count-convirt-drbd:1
  653. Nov 1 03:14:39 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin02-fail-count-convirt-drbd:1, magic=NA, cib=0.28.4) : Transient attribute: update
  654. Nov 1 03:14:39 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin02-last-failure-convirt-drbd:1, magic=NA, cib=0.28.5) : Transient attribute: update
  655. Nov 1 03:14:39 admin01 attrd: [15347]: info: find_hash_entry: Creating hash entry for last-failure-convirt-drbd:1
  656. Nov 1 03:14:39 admin01 crm_attribute: [15950]: info: determine_host: Mapped admin01 to admin01
  657. Nov 1 03:14:39 admin01 crm_attribute: [15950]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  658. Nov 1 03:14:39 admin01 crm_attribute: [15950]: info: main: Update master-sitescope-drbd:1=5 sent via attrd
  659. Nov 1 03:14:39 admin01 crm_attribute: [15950]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  660. Nov 1 03:14:39 admin01 crm_attribute: [15953]: info: determine_host: Mapped admin01 to admin01
  661. Nov 1 03:14:39 admin01 crm_attribute: [15953]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  662. Nov 1 03:14:39 admin01 crm_attribute: [15953]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  663. Nov 1 03:14:39 admin01 crm_attribute: [15953]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  664. Nov 1 03:14:39 admin01 lrmd: [15346]: info: RA output: (sitescope-drbd:1:notify:stdout)
  665. Nov 1 03:14:39 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:notify process 15892 exited with return code 0.
  666. Nov 1 03:14:39 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_notify_0 (call=13, rc=0, cib-update=42, confirmed=true) ok
  667. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_post_notify_promote_0 (109) confirmed on admin01 (rc=0)
  668. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_post_notify_promote_0 (107) confirmed on admin02 (rc=0)
  669. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 71 fired and confirmed
  670. Nov 1 03:14:39 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  671. Nov 1 03:14:39 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 15895 exited with return code 0.
  672. Nov 1 03:14:39 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=14, rc=0, cib-update=43, confirmed=true) ok
  673. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_promote_0 (91) confirmed on admin01 (rc=0)
  674. Nov 1 03:14:39 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_post_notify_promote_0 (93) confirmed on admin02 (rc=0)
  675. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
  676. Nov 1 03:14:39 admin01 crmd: [15349]: info: run_graph: ====================================================
  677. Nov 1 03:14:39 admin01 crmd: [15349]: notice: run_graph: Transition 2 (Complete=22, Pending=0, Fired=0, Skipped=8, Incomplete=0, Source=/var/lib/pengine/pe-input-2700.bz2): Stopped
  678. Nov 1 03:14:39 admin01 crmd: [15349]: info: te_graph_trigger: Transition 2 is now complete
  679. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  680. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  681. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_pe_invoke: Query 44: Requesting the current CIB: S_POLICY_ENGINE
  682. Nov 1 03:14:39 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=44, ref=pe_calc-dc-1320117279-44, seq=320, quorate=1
  683. Nov 1 03:14:39 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  684. Nov 1 03:14:39 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  685. Nov 1 03:14:39 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  686. Nov 1 03:14:39 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  687. Nov 1 03:14:39 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  688. Nov 1 03:14:39 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-drbd:1_promote_0 on admin02: unknown error (1)
  689. Nov 1 03:14:39 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  690. Nov 1 03:14:39 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_stop_0 on admin01: unknown error (1)
  691. Nov 1 03:14:39 admin01 pengine: [15348]: info: native_add_running: resource convirt-kvm isnt managed
  692. Nov 1 03:14:39 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 (unmanaged) FAILED
  693. Nov 1 03:14:39 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  694. Nov 1 03:14:39 admin01 pengine: [15348]: notice: native_print: convirt-drbd:1#011(ocf::linbit:drbd):#011Master admin02 FAILED
  695. Nov 1 03:14:39 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  696. Nov 1 03:14:39 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Stopped
  697. Nov 1 03:14:39 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  698. Nov 1 03:14:39 admin01 pengine: [15348]: notice: short_print: Masters: [ admin02 ]
  699. Nov 1 03:14:39 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  700. Nov 1 03:14:39 admin01 pengine: [15348]: info: get_failcount: convirt-kvm has failed INFINITY times on admin01
  701. Nov 1 03:14:39 admin01 pengine: [15348]: WARN: common_apply_stickiness: Forcing convirt-kvm away from admin01 after 1000000 failures (max=1000000)
  702. Nov 1 03:14:39 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  703. Nov 1 03:14:39 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  704. Nov 1 03:14:39 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  705. Nov 1 03:14:39 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  706. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 0 instances of a possible 1 to master
  707. Nov 1 03:14:39 admin01 pengine: [15348]: info: native_color: Unmanaged resource convirt-kvm allocated to 'nowhere': failed
  708. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 0 instances of a possible 1 to master
  709. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  710. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  711. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  712. Nov 1 03:14:39 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  713. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  714. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  715. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  716. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  717. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  718. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  719. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  720. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  721. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-kvm on admin02
  722. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-drbd:0 on admin02
  723. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  724. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  725. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-drbd:0 on admin02
  726. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for sitescope-drbd:1 on admin01
  727. Nov 1 03:14:39 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for sitescope-drbd:1 on admin01
  728. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Leave convirt-kvm#011(Started unmanaged)
  729. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:0#011(Slave admin01)
  730. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Demote convirt-drbd:1#011(Master -> Slave admin02)
  731. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Recover convirt-drbd:1#011(Master admin02)
  732. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Start sitescope-kvm#011(admin02)
  733. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:0#011(Master admin02)
  734. Nov 1 03:14:39 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:1#011(Slave admin01)
  735. Nov 1 03:14:39 admin01 pengine: [15348]: WARN: should_dump_input: Ignoring requirement that convirt-kvm_stop_0 comeplete before ms-convirt_demote_0: unmanaged failed resources cannot prevent clone shutdown
  736. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  737. Nov 1 03:14:40 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 3: 41 actions in 41 synapses
  738. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1320117279-44) derived from /var/lib/pengine/pe-input-2701.bz2
  739. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
  740. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 42: start sitescope-kvm_start_0 on admin02
  741. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 48: monitor sitescope-drbd:0_monitor_10000 on admin02
  742. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 53: monitor sitescope-drbd:1_monitor_30000 on admin01 (local)
  743. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=53:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_monitor_30000 )
  744. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:15: monitor
  745. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 52: monitor sitescope-drbd:1_monitor_120000 on admin01 (local)
  746. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=52:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=sitescope-drbd:1_monitor_120000 )
  747. Nov 1 03:14:40 admin01 lrmd: [15346]: info: perform_op:2884: operation monitor[15] on ocf::drbd::sitescope-drbd:1 for client 15349, its parameters: CRM_meta_clone=[1] CRM_meta_timeout=[20000] CRM_meta_notify_slave_resource=[ ] CRM_meta_notify_active_resource=[ ] CRM_meta_notify_demote_uname=[ ] drbd_resource=[sitescope] CRM_meta_notify_inactive_resource=[sitescope-drbd:0 sitescope-drbd:1 ] CRM_meta_master_node_max=[1] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_master_resource=[ ] CRM_meta_clone_node_max=[1] CRM_meta_notify_demote_resource=[ ] CRM_meta_clone_m for rsc is already running.
  748. Nov 1 03:14:40 admin01 lrmd: [15346]: info: perform_op:2894: postponing all ops on resource sitescope-drbd:1 by 1000 ms
  749. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 89: notify convirt-drbd:0_pre_notify_demote_0 on admin01 (local)
  750. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=89:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  751. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:17: notify
  752. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 91: notify convirt-drbd:1_pre_notify_demote_0 on admin02
  753. Nov 1 03:14:40 admin01 pengine: [15348]: info: process_pe_message: Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-2701.bz2
  754. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 15971 exited with return code 0.
  755. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=17, rc=0, cib-update=45, confirmed=true) ok
  756. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_pre_notify_demote_0 (89) confirmed on admin01 (rc=0)
  757. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_pre_notify_demote_0 (91) confirmed on admin02 (rc=0)
  758. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
  759. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
  760. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 14: demote convirt-drbd:1_demote_0 on admin02
  761. Nov 1 03:14:40 admin01 crm_attribute: [16022]: info: determine_host: Mapped admin01 to admin01
  762. Nov 1 03:14:40 admin01 crm_attribute: [16022]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  763. Nov 1 03:14:40 admin01 crm_attribute: [16022]: info: main: Update master-sitescope-drbd:1=5 sent via attrd
  764. Nov 1 03:14:40 admin01 crm_attribute: [16022]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  765. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:0_monitor_10000 (48) confirmed on admin02 (rc=0)
  766. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:monitor process 15970 exited with return code 0.
  767. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:sitescope-drbd:1:16: monitor
  768. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_monitor_30000 (call=15, rc=0, cib-update=46, confirmed=false) ok
  769. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_monitor_30000 (53) confirmed on admin01 (rc=0)
  770. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_demote_0 (14) confirmed on admin02 (rc=0)
  771. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
  772. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 40 fired and confirmed
  773. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 90: notify convirt-drbd:0_post_notify_demote_0 on admin01 (local)
  774. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=90:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  775. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:18: notify
  776. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 92: notify convirt-drbd:1_post_notify_demote_0 on admin02
  777. Nov 1 03:14:40 admin01 crm_attribute: [16076]: info: determine_host: Mapped admin01 to admin01
  778. Nov 1 03:14:40 admin01 crm_attribute: [16076]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  779. Nov 1 03:14:40 admin01 crm_attribute: [16076]: info: main: Update master-sitescope-drbd:1=5 sent via attrd
  780. Nov 1 03:14:40 admin01 crm_attribute: [16076]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  781. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed sitescope-drbd:1:monitor process 16031 exited with return code 0.
  782. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation sitescope-drbd:1_monitor_120000 (call=16, rc=0, cib-update=47, confirmed=false) ok
  783. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-drbd:1_monitor_120000 (52) confirmed on admin01 (rc=0)
  784. Nov 1 03:14:40 admin01 crm_attribute: [16103]: info: determine_host: Mapped admin01 to admin01
  785. Nov 1 03:14:40 admin01 crm_attribute: [16103]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  786. Nov 1 03:14:40 admin01 crm_attribute: [16103]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  787. Nov 1 03:14:40 admin01 crm_attribute: [16103]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  788. Nov 1 03:14:40 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  789. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16042 exited with return code 0.
  790. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=18, rc=0, cib-update=48, confirmed=true) ok
  791. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_demote_0 (90) confirmed on admin01 (rc=0)
  792. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_post_notify_demote_0 (92) confirmed on admin02 (rc=0)
  793. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 41 fired and confirmed
  794. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
  795. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 86: notify convirt-drbd:0_pre_notify_stop_0 on admin01 (local)
  796. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=86:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  797. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:19: notify
  798. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 88: notify convirt-drbd:1_pre_notify_stop_0 on admin02
  799. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16112 exited with return code 0.
  800. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=19, rc=0, cib-update=49, confirmed=true) ok
  801. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_pre_notify_stop_0 (86) confirmed on admin01 (rc=0)
  802. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_pre_notify_stop_0 (88) confirmed on admin02 (rc=0)
  803. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
  804. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
  805. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 1: stop convirt-drbd:1_stop_0 on admin02
  806. Nov 1 03:14:40 admin01 kernel: d-con convirt: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
  807. Nov 1 03:14:40 admin01 kernel: d-con convirt: asender terminated
  808. Nov 1 03:14:40 admin01 kernel: d-con convirt: Terminating asender thread
  809. Nov 1 03:14:40 admin01 kernel: d-con convirt: Connection closed
  810. Nov 1 03:14:40 admin01 kernel: d-con convirt: conn( TearDown -> Unconnected )
  811. Nov 1 03:14:40 admin01 kernel: d-con convirt: receiver terminated
  812. Nov 1 03:14:40 admin01 kernel: d-con convirt: Restarting receiver thread
  813. Nov 1 03:14:40 admin01 kernel: d-con convirt: receiver (re)started
  814. Nov 1 03:14:40 admin01 kernel: d-con convirt: conn( Unconnected -> WFConnection )
  815. Nov 1 03:14:40 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:163 - Triggered transition abort (complete=0, tag=transient_attributes, id=admin02, magic=NA, cib=0.28.20) : Transient attribute: removal
  816. Nov 1 03:14:40 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
  817. Nov 1 03:14:40 admin01 crmd: [15349]: info: update_abort_priority: Abort action done superceeded by restart
  818. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-kvm_start_0 (42) confirmed on admin02 (rc=0)
  819. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_stop_0 (1) confirmed on admin02 (rc=0)
  820. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 25 fired and confirmed
  821. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
  822. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 87: notify convirt-drbd:0_post_notify_stop_0 on admin01 (local)
  823. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=87:3:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  824. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:20: notify
  825. Nov 1 03:14:40 admin01 crm_attribute: [16164]: info: determine_host: Mapped admin01 to admin01
  826. Nov 1 03:14:40 admin01 crm_attribute: [16164]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  827. Nov 1 03:14:40 admin01 crm_attribute: [16164]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  828. Nov 1 03:14:40 admin01 crm_attribute: [16164]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  829. Nov 1 03:14:40 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  830. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16134 exited with return code 0.
  831. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=20, rc=0, cib-update=50, confirmed=true) ok
  832. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_stop_0 (87) confirmed on admin01 (rc=0)
  833. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
  834. Nov 1 03:14:40 admin01 crmd: [15349]: info: run_graph: ====================================================
  835. Nov 1 03:14:40 admin01 crmd: [15349]: notice: run_graph: Transition 3 (Complete=25, Pending=0, Fired=0, Skipped=11, Incomplete=5, Source=/var/lib/pengine/pe-input-2701.bz2): Stopped
  836. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_graph_trigger: Transition 3 is now complete
  837. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  838. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  839. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_pe_invoke: Query 51: Requesting the current CIB: S_POLICY_ENGINE
  840. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=51, ref=pe_calc-dc-1320117280-58, seq=320, quorate=1
  841. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  842. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  843. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  844. Nov 1 03:14:40 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  845. Nov 1 03:14:40 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  846. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-drbd:1_promote_0 on admin02: unknown error (1)
  847. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  848. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_stop_0 on admin01: unknown error (1)
  849. Nov 1 03:14:40 admin01 pengine: [15348]: info: native_add_running: resource convirt-kvm isnt managed
  850. Nov 1 03:14:40 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 (unmanaged) FAILED
  851. Nov 1 03:14:40 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  852. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  853. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Stopped: [ convirt-drbd:1 ]
  854. Nov 1 03:14:40 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin02
  855. Nov 1 03:14:40 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  856. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Masters: [ admin02 ]
  857. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  858. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: convirt-kvm has failed INFINITY times on admin01
  859. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: common_apply_stickiness: Forcing convirt-kvm away from admin01 after 1000000 failures (max=1000000)
  860. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  861. Nov 1 03:14:40 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  862. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  863. Nov 1 03:14:40 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  864. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 0 instances of a possible 1 to master
  865. Nov 1 03:14:40 admin01 pengine: [15348]: info: native_color: Unmanaged resource convirt-kvm allocated to 'nowhere': failed
  866. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 0 instances of a possible 1 to master
  867. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  868. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  869. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  870. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  871. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  872. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  873. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  874. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  875. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  876. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  877. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:1 on admin02
  878. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:1 on admin02
  879. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for sitescope-kvm on admin02
  880. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave convirt-kvm#011(Started unmanaged)
  881. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:0#011(Slave admin01)
  882. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Start convirt-drbd:1#011(admin02)
  883. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-kvm#011(Started admin02)
  884. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:0#011(Master admin02)
  885. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:1#011(Slave admin01)
  886. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  887. Nov 1 03:14:40 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 4: 15 actions in 15 synapses
  888. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1320117280-58) derived from /var/lib/pengine/pe-input-2702.bz2
  889. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
  890. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 45: monitor sitescope-kvm_monitor_10000 on admin02
  891. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 83: notify convirt-drbd:0_pre_notify_start_0 on admin01 (local)
  892. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=83:4:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  893. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:21: notify
  894. Nov 1 03:14:40 admin01 pengine: [15348]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-2702.bz2
  895. Nov 1 03:14:40 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  896. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16173 exited with return code 0.
  897. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=21, rc=0, cib-update=52, confirmed=true) ok
  898. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_pre_notify_start_0 (83) confirmed on admin01 (rc=0)
  899. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
  900. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
  901. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 16: start convirt-drbd:1_start_0 on admin02
  902. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action sitescope-kvm_monitor_10000 (45) confirmed on admin02 (rc=0)
  903. Nov 1 03:14:40 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:149 - Triggered transition abort (complete=0, tag=nvpair, id=status-admin02-master-convirt-drbd:1, magic=NA, cib=0.28.27) : Transient attribute: update
  904. Nov 1 03:14:40 admin01 crmd: [15349]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
  905. Nov 1 03:14:40 admin01 crmd: [15349]: info: update_abort_priority: Abort action done superceeded by restart
  906. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_start_0 (16) confirmed on admin02 (rc=0)
  907. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
  908. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
  909. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 84: notify convirt-drbd:0_post_notify_start_0 on admin01 (local)
  910. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=84:4:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  911. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:22: notify
  912. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 85: notify convirt-drbd:1_post_notify_start_0 on admin02
  913. Nov 1 03:14:40 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  914. Nov 1 03:14:40 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16213 exited with return code 0.
  915. Nov 1 03:14:40 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=22, rc=0, cib-update=53, confirmed=true) ok
  916. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_start_0 (84) confirmed on admin01 (rc=0)
  917. Nov 1 03:14:40 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_post_notify_start_0 (85) confirmed on admin02 (rc=0)
  918. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
  919. Nov 1 03:14:40 admin01 crmd: [15349]: info: run_graph: ====================================================
  920. Nov 1 03:14:40 admin01 crmd: [15349]: notice: run_graph: Transition 4 (Complete=11, Pending=0, Fired=0, Skipped=4, Incomplete=0, Source=/var/lib/pengine/pe-input-2702.bz2): Stopped
  921. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_graph_trigger: Transition 4 is now complete
  922. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  923. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  924. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_pe_invoke: Query 54: Requesting the current CIB: S_POLICY_ENGINE
  925. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=54, ref=pe_calc-dc-1320117280-64, seq=320, quorate=1
  926. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  927. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  928. Nov 1 03:14:40 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  929. Nov 1 03:14:40 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  930. Nov 1 03:14:40 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  931. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-drbd:1_promote_0 on admin02: unknown error (1)
  932. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  933. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_stop_0 on admin01: unknown error (1)
  934. Nov 1 03:14:40 admin01 pengine: [15348]: info: native_add_running: resource convirt-kvm isnt managed
  935. Nov 1 03:14:40 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 (unmanaged) FAILED
  936. Nov 1 03:14:40 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  937. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 admin02 ]
  938. Nov 1 03:14:40 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin02
  939. Nov 1 03:14:40 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  940. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Masters: [ admin02 ]
  941. Nov 1 03:14:40 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  942. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: convirt-kvm has failed INFINITY times on admin01
  943. Nov 1 03:14:40 admin01 pengine: [15348]: WARN: common_apply_stickiness: Forcing convirt-kvm away from admin01 after 1000000 failures (max=1000000)
  944. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  945. Nov 1 03:14:40 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  946. Nov 1 03:14:40 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  947. Nov 1 03:14:40 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  948. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Slave admin02)
  949. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  950. Nov 1 03:14:40 admin01 pengine: [15348]: info: native_color: Unmanaged resource convirt-kvm allocated to 'nowhere': failed
  951. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Slave admin02)
  952. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  953. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  954. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  955. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  956. Nov 1 03:14:40 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  957. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  958. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  959. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:1 on admin02
  960. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (120s) for convirt-drbd:0 on admin01
  961. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (30s) for convirt-drbd:0 on admin01
  962. Nov 1 03:14:40 admin01 pengine: [15348]: notice: RecurringOp: Start recurring monitor (10s) for convirt-drbd:1 on admin02
  963. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave convirt-kvm#011(Started unmanaged)
  964. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:0#011(Slave admin01)
  965. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Promote convirt-drbd:1#011(Slave -> Master admin02)
  966. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-kvm#011(Started admin02)
  967. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:0#011(Master admin02)
  968. Nov 1 03:14:40 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:1#011(Slave admin01)
  969. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  970. Nov 1 03:14:40 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 5: 14 actions in 14 synapses
  971. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1320117280-64) derived from /var/lib/pengine/pe-input-2703.bz2
  972. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
  973. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 91: notify convirt-drbd:0_pre_notify_promote_0 on admin01 (local)
  974. Nov 1 03:14:40 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=91:5:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  975. Nov 1 03:14:40 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:23: notify
  976. Nov 1 03:14:40 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 93: notify convirt-drbd:1_pre_notify_promote_0 on admin02
  977. Nov 1 03:14:40 admin01 pengine: [15348]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-2703.bz2
  978. Nov 1 03:14:41 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16253 exited with return code 0.
  979. Nov 1 03:14:41 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=23, rc=0, cib-update=55, confirmed=true) ok
  980. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_pre_notify_promote_0 (91) confirmed on admin01 (rc=0)
  981. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_pre_notify_promote_0 (93) confirmed on admin02 (rc=0)
  982. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
  983. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
  984. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 20: promote convirt-drbd:1_promote_0 on admin02
  985. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_promote_0 (20) confirmed on admin02 (rc=0)
  986. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
  987. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
  988. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 92: notify convirt-drbd:0_post_notify_promote_0 on admin01 (local)
  989. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=92:5:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_notify_0 )
  990. Nov 1 03:14:41 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:24: notify
  991. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 94: notify convirt-drbd:1_post_notify_promote_0 on admin02
  992. Nov 1 03:14:41 admin01 crm_attribute: [16305]: info: determine_host: Mapped admin01 to admin01
  993. Nov 1 03:14:41 admin01 crm_attribute: [16305]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  994. Nov 1 03:14:41 admin01 crm_attribute: [16305]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  995. Nov 1 03:14:41 admin01 crm_attribute: [16305]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  996. Nov 1 03:14:41 admin01 lrmd: [15346]: info: RA output: (convirt-drbd:0:notify:stdout)
  997. Nov 1 03:14:41 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:notify process 16275 exited with return code 0.
  998. Nov 1 03:14:41 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_notify_0 (call=24, rc=0, cib-update=56, confirmed=true) ok
  999. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_post_notify_promote_0 (92) confirmed on admin01 (rc=0)
  1000. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_post_notify_promote_0 (94) confirmed on admin02 (rc=0)
  1001. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
  1002. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 16: monitor convirt-drbd:0_monitor_30000 on admin01 (local)
  1003. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=16:5:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_monitor_30000 )
  1004. Nov 1 03:14:41 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:25: monitor
  1005. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 15: monitor convirt-drbd:0_monitor_120000 on admin01 (local)
  1006. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_lrm_rsc_op: Performing key=15:5:0:c5a13ba5-3b1f-4b81-82a2-838830e194a8 op=convirt-drbd:0_monitor_120000 )
  1007. Nov 1 03:14:41 admin01 lrmd: [15346]: info: perform_op:2884: operation monitor[25] on ocf::drbd::convirt-drbd:0 for client 15349, its parameters: CRM_meta_clone=[0] CRM_meta_timeout=[20000] CRM_meta_notify_slave_resource=[ ] CRM_meta_notify_active_resource=[ ] CRM_meta_notify_demote_uname=[ ] drbd_resource=[convirt] CRM_meta_notify_inactive_resource=[convirt-drbd:0 convirt-drbd:1 ] CRM_meta_master_node_max=[1] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_master_resource=[ ] CRM_meta_clone_node_max=[1] CRM_meta_notify_demote_resource=[ ] CRM_meta_clone_max=[2] C for rsc is already running.
  1008. Nov 1 03:14:41 admin01 lrmd: [15346]: info: perform_op:2894: postponing all ops on resource convirt-drbd:0 by 1000 ms
  1009. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_rsc_command: Initiating action 21: monitor convirt-drbd:1_monitor_10000 on admin02
  1010. Nov 1 03:14:41 admin01 crm_attribute: [16344]: info: determine_host: Mapped admin01 to admin01
  1011. Nov 1 03:14:41 admin01 crm_attribute: [16344]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  1012. Nov 1 03:14:41 admin01 crm_attribute: [16344]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  1013. Nov 1 03:14:41 admin01 crm_attribute: [16344]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  1014. Nov 1 03:14:41 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:monitor process 16314 exited with return code 0.
  1015. Nov 1 03:14:41 admin01 lrmd: [15346]: info: rsc:convirt-drbd:0:26: monitor
  1016. Nov 1 03:14:41 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_monitor_30000 (call=25, rc=0, cib-update=57, confirmed=false) ok
  1017. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_monitor_30000 (16) confirmed on admin01 (rc=0)
  1018. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:1_monitor_10000 (21) confirmed on admin02 (rc=0)
  1019. Nov 1 03:14:41 admin01 kernel: d-con convirt: Handshake successful: Agreed network protocol version 100
  1020. Nov 1 03:14:41 admin01 kernel: d-con convirt: conn( WFConnection -> WFReportParams )
  1021. Nov 1 03:14:41 admin01 kernel: d-con convirt: Starting asender thread (from drbd_r_convirt [15699])
  1022. Nov 1 03:14:41 admin01 kernel: block drbd0: drbd_sync_handshake:
  1023. Nov 1 03:14:41 admin01 kernel: block drbd0: self B21A3767BB62B1E0:0000000000000000:C9D11CAB529FACB6:C9D01CAB529FACB6 bits:0 flags:0
  1024. Nov 1 03:14:41 admin01 kernel: block drbd0: peer B2DBA3F84F5FA3F9:B21A3767BB62B1E0:C9D11CAB529FACB6:C9D01CAB529FACB6 bits:0 flags:0
  1025. Nov 1 03:14:41 admin01 kernel: block drbd0: uuid_compare()=-1 by rule 50
  1026. Nov 1 03:14:41 admin01 kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )
  1027. Nov 1 03:14:41 admin01 kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
  1028. Nov 1 03:14:41 admin01 kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
  1029. Nov 1 03:14:41 admin01 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
  1030. Nov 1 03:14:41 admin01 kernel: block drbd0: updated sync uuid B21B3767BB62B1E0:0000000000000000:C9D11CAB529FACB6:C9D01CAB529FACB6
  1031. Nov 1 03:14:41 admin01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
  1032. Nov 1 03:14:41 admin01 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
  1033. Nov 1 03:14:41 admin01 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
  1034. Nov 1 03:14:41 admin01 kernel: block drbd0: Began resync as SyncTarget (will sync 0 KB [0 bits set]).
  1035. Nov 1 03:14:41 admin01 kernel: block drbd0: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
  1036. Nov 1 03:14:41 admin01 kernel: block drbd0: updated UUIDs B2DBA3F84F5FA3F8:0000000000000000:B21B3767BB62B1E0:B21A3767BB62B1E0
  1037. Nov 1 03:14:41 admin01 kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
  1038. Nov 1 03:14:41 admin01 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0
  1039. Nov 1 03:14:41 admin01 crm-unfence-peer.sh[16379]: invoked for convirt
  1040. Nov 1 03:14:41 admin01 crm_attribute: [16393]: info: determine_host: Mapped admin01 to admin01
  1041. Nov 1 03:14:41 admin01 crm_attribute: [16393]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
  1042. Nov 1 03:14:41 admin01 crm_attribute: [16393]: info: main: Update master-convirt-drbd:0=5 sent via attrd
  1043. Nov 1 03:14:41 admin01 crm_attribute: [16393]: info: crm_xml_cleanup: Cleaning up memory from libxml2
  1044. Nov 1 03:14:41 admin01 lrmd: [15346]: info: Managed convirt-drbd:0:monitor process 16353 exited with return code 0.
  1045. Nov 1 03:14:41 admin01 crmd: [15349]: info: process_lrm_event: LRM operation convirt-drbd:0_monitor_120000 (call=26, rc=0, cib-update=58, confirmed=false) ok
  1046. Nov 1 03:14:41 admin01 crmd: [15349]: info: match_graph_event: Action convirt-drbd:0_monitor_120000 (15) confirmed on admin01 (rc=0)
  1047. Nov 1 03:14:41 admin01 crmd: [15349]: info: run_graph: ====================================================
  1048. Nov 1 03:14:41 admin01 crmd: [15349]: notice: run_graph: Transition 5 (Complete=14, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-2703.bz2): Complete
  1049. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_graph_trigger: Transition 5 is now complete
  1050. Nov 1 03:14:41 admin01 crmd: [15349]: info: notify_crmd: Transition 5 status: done - <null>
  1051. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  1052. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: Starting PEngine Recheck Timer
  1053. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <cib admin_epoch="0" epoch="28" num_updates="38" >
  1054. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <configuration >
  1055. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <constraints >
  1056. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <rsc_location rsc="ms-convirt" id="drbd-fence-by-handler-ms-convirt" __crm_diff_marker__="removed:top" >
  1057. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-rule-ms-convirt" >
  1058. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- <expression attribute="#uname" operation="ne" value="admin02" id="drbd-fence-by-handler-expr-ms-convirt" />
  1059. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- </rule>
  1060. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- </rsc_location>
  1061. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- </constraints>
  1062. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- </configuration>
  1063. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff- </cib>
  1064. Nov 1 03:14:41 admin01 cib: [15345]: info: cib:diff+ <cib epoch="29" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" have-quorum="1" cib-last-written="Tue Nov 1 02:43:42 2011" dc-uuid="admin01" />
  1065. Nov 1 03:14:41 admin01 cib: [15345]: info: cib_process_request: Operation complete: op cib_delete for section 'all' (origin=local/cibadmin/2, version=0.29.1): ok (rc=0)
  1066. Nov 1 03:14:41 admin01 crmd: [15349]: info: abort_transition_graph: te_update_diff:131 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.29.1) : Non-status change
  1067. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  1068. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
  1069. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_pe_invoke: Query 59: Requesting the current CIB: S_POLICY_ENGINE
  1070. Nov 1 03:14:41 admin01 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
  1071. Nov 1 03:14:41 admin01 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  1072. Nov 1 03:14:41 admin01 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  1073. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_pe_invoke_callback: Invoking the PE: query=59, ref=pe_calc-dc-1320117281-73, seq=320, quorate=1
  1074. Nov 1 03:14:41 admin01 pengine: [15348]: info: unpack_config: Startup probes: enabled
  1075. Nov 1 03:14:41 admin01 pengine: [15348]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
  1076. Nov 1 03:14:41 admin01 pengine: [15348]: info: unpack_domains: Unpacking domains
  1077. Nov 1 03:14:41 admin01 pengine: [15348]: info: determine_online_status: Node admin02 is online
  1078. Nov 1 03:14:41 admin01 pengine: [15348]: info: determine_online_status: Node admin01 is online
  1079. Nov 1 03:14:41 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_monitor_0 on admin01: unknown error (1)
  1080. Nov 1 03:14:41 admin01 pengine: [15348]: WARN: unpack_rsc_op: Processing failed op convirt-kvm_stop_0 on admin01: unknown error (1)
  1081. Nov 1 03:14:41 admin01 pengine: [15348]: info: native_add_running: resource convirt-kvm isnt managed
  1082. Nov 1 03:14:41 admin01 pengine: [15348]: notice: native_print: convirt-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin01 (unmanaged) FAILED
  1083. Nov 1 03:14:41 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-convirt [convirt-drbd]
  1084. Nov 1 03:14:41 admin01 pengine: [15348]: notice: short_print: Masters: [ admin02 ]
  1085. Nov 1 03:14:41 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  1086. Nov 1 03:14:41 admin01 pengine: [15348]: notice: native_print: sitescope-kvm#011(ocf::heartbeat:VirtualDomain):#011Started admin02
  1087. Nov 1 03:14:41 admin01 pengine: [15348]: notice: clone_print: Master/Slave Set: ms-sitescope [sitescope-drbd]
  1088. Nov 1 03:14:41 admin01 pengine: [15348]: notice: short_print: Masters: [ admin02 ]
  1089. Nov 1 03:14:41 admin01 pengine: [15348]: notice: short_print: Slaves: [ admin01 ]
  1090. Nov 1 03:14:41 admin01 pengine: [15348]: info: get_failcount: convirt-kvm has failed INFINITY times on admin01
  1091. Nov 1 03:14:41 admin01 pengine: [15348]: WARN: common_apply_stickiness: Forcing convirt-kvm away from admin01 after 1000000 failures (max=1000000)
  1092. Nov 1 03:14:41 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  1093. Nov 1 03:14:41 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  1094. Nov 1 03:14:41 admin01 pengine: [15348]: info: get_failcount: ms-convirt has failed 1 times on admin02
  1095. Nov 1 03:14:41 admin01 pengine: [15348]: notice: common_apply_stickiness: ms-convirt can fail 999999 more times on admin02 before being forced off
  1096. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Master admin02)
  1097. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  1098. Nov 1 03:14:41 admin01 pengine: [15348]: info: native_color: Unmanaged resource convirt-kvm allocated to 'nowhere': failed
  1099. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: Promoting convirt-drbd:1 (Master admin02)
  1100. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: ms-convirt: Promoted 1 instances of a possible 1 to master
  1101. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  1102. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  1103. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: Promoting sitescope-drbd:0 (Master admin02)
  1104. Nov 1 03:14:41 admin01 pengine: [15348]: info: master_color: ms-sitescope: Promoted 1 instances of a possible 1 to master
  1105. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave convirt-kvm#011(Started unmanaged)
  1106. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:0#011(Slave admin01)
  1107. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave convirt-drbd:1#011(Master admin02)
  1108. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-kvm#011(Started admin02)
  1109. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:0#011(Master admin02)
  1110. Nov 1 03:14:41 admin01 pengine: [15348]: notice: LogActions: Leave sitescope-drbd:1#011(Slave admin01)
  1111. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  1112. Nov 1 03:14:41 admin01 crmd: [15349]: info: unpack_graph: Unpacked transition 6: 0 actions in 0 synapses
  1113. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1320117281-73) derived from /var/lib/pengine/pe-input-2704.bz2
  1114. Nov 1 03:14:41 admin01 crmd: [15349]: info: run_graph: ====================================================
  1115. Nov 1 03:14:41 admin01 crmd: [15349]: notice: run_graph: Transition 6 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-2704.bz2): Complete
  1116. Nov 1 03:14:41 admin01 crmd: [15349]: info: te_graph_trigger: Transition 6 is now complete
  1117. Nov 1 03:14:41 admin01 crmd: [15349]: info: notify_crmd: Transition 6 status: done - <null>
  1118. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  1119. Nov 1 03:14:41 admin01 crmd: [15349]: info: do_state_transition: Starting PEngine Recheck Timer
  1120. Nov 1 03:14:41 admin01 pengine: [15348]: info: process_pe_message: Transition 6: PEngine Input stored in: /var/lib/pengine/pe-input-2704.bz2
  1121. Nov 1 03:14:41 admin01 cib: [15345]: info: Managed write_cib_contents process 16411 exited with return code 0.
  1122.  
  1123.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!