API tools faq
paste
Advertisement
Guest User

PacemakerDRBDfailover

a guest
Sep 6th, 2016
155
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 12.18 KB | None | 0 0
raw download clone embed print report
  1. Sep 6 14:50:18 node2 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  2. Sep 6 14:51:18 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (10000)
  3. Sep 6 14:51:18 node2 attrd[1711]: notice: Sent update 197: master-ClusterDB=10000
  4. Sep 6 14:51:18 node2 crmd[1713]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  5. Sep 6 14:51:18 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  6. Sep 6 14:51:18 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  7. Sep 6 14:51:18 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  8. Sep 6 14:51:18 node2 crmd[1713]: notice: Transition 41 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-149.bz2): Complete
  9. Sep 6 14:51:18 node2 crmd[1713]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  10. Sep 6 14:51:18 node2 pengine[1712]: notice: Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-149.bz2
  11. Sep 6 14:53:33 node2 crmd[1713]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  12. Sep 6 14:53:33 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  13. Sep 6 14:53:33 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  14. Sep 6 14:53:33 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  15. Sep 6 14:53:33 node2 pengine[1712]: notice: Move Cluster_VIP#011(Started node1 -> node2)
  16. Sep 6 14:53:33 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Stopped node1)
  17. Sep 6 14:53:33 node2 pengine[1712]: notice: Promote ClusterDB:1#011(Slave -> Master node2)
  18. Sep 6 14:53:33 node2 pengine[1712]: notice: Stop ClusterFS#011(node1)
  19. Sep 6 14:53:33 node2 pengine[1712]: notice: Move WebSite#011(Started node1 -> node2)
  20. Sep 6 14:53:33 node2 pengine[1712]: notice: Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-150.bz2
  21. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 6: stop Cluster_VIP_stop_0 on node1
  22. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 4: cancel ClusterDB_monitor_60000 on node2 (local)
  23. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 38: stop ClusterFS_stop_0 on node1
  24. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 54: notify ClusterDB_pre_notify_demote_0 on node2 (local)
  25. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=170, rc=0, cib-update=0, confirmed=true)
  26. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 7: start Cluster_VIP_start_0 on node2 (local)
  27. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 52: notify ClusterDB_pre_notify_demote_0 on node1
  28. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: Adding inet address 10.0.2.200/24 with broadcast address 10.0.2.255 to device eth2
  29. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: Bringing device eth2 up
  30. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.2.200 eth2 10.0.2.200 auto not_used not_used
  31. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation Cluster_VIP_start_0: ok (node=node2, call=171, rc=0, cib-update=251, confirmed=true)
  32. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 8: monitor Cluster_VIP_monitor_10000 on node2 (local)
  33. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 39: stop WebSite_stop_0 on node1
  34. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 9: demote ClusterDB_demote_0 on node1
  35. Sep 6 14:53:33 node2 kernel: block drbd0: peer( Primary -> Secondary )
  36. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 53: notify ClusterDB_post_notify_demote_0 on node1
  37. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 55: notify ClusterDB_post_notify_demote_0 on node2 (local)
  38. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=173, rc=0, cib-update=0, confirmed=true)
  39. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 47: notify ClusterDB_pre_notify_stop_0 on node1
  40. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 48: notify ClusterDB_pre_notify_stop_0 on node2 (local)
  41. Sep 6 14:53:34 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=174, rc=0, cib-update=0, confirmed=true)
  42. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 10: stop ClusterDB_stop_0 on node1
  43. Sep 6 14:53:34 node2 kernel: block drbd0: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
  44. Sep 6 14:53:34 node2 kernel: block drbd0: meta connection shut down by peer.
  45. Sep 6 14:53:34 node2 kernel: block drbd0: asender terminated
  46. Sep 6 14:53:34 node2 kernel: block drbd0: Terminating drbd0_asender
  47. Sep 6 14:53:34 node2 kernel: block drbd0: Connection closed
  48. Sep 6 14:53:34 node2 kernel: block drbd0: conn( TearDown -> Unconnected )
  49. Sep 6 14:53:34 node2 kernel: block drbd0: receiver terminated
  50. Sep 6 14:53:34 node2 kernel: block drbd0: Restarting drbd0_receiver
  51. Sep 6 14:53:34 node2 kernel: block drbd0: receiver (re)started
  52. Sep 6 14:53:34 node2 kernel: block drbd0: conn( Unconnected -> WFConnection )
  53. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 49: notify ClusterDB_post_notify_stop_0 on node2 (local)
  54. Sep 6 14:53:34 node2 crmd[1713]: warning: No match for shutdown action on node1
  55. Sep 6 14:53:34 node2 crmd[1713]: notice: Transition aborted by deletion of nvpair[@id='status-node1-master-ClusterDB']: Transient attribute change (cib=0.102.8, source=abort_unless_down:343, path=/cib/status/node_state[@id='node1']/transient_attributes[@id='node1']/instance_attributes[@id='status-node1']/nvpair[@id='status-node1-master-ClusterDB'], 0)
  56. Sep 6 14:53:34 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (1000)
  57. Sep 6 14:53:34 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=175, rc=0, cib-update=0, confirmed=true)
  58. Sep 6 14:53:34 node2 attrd[1711]: notice: Sent update 201: master-ClusterDB=1000
  59. Sep 6 14:53:35 node2 crmd[1713]: notice: Transition 42 (Complete=29, Pending=0, Fired=0, Skipped=2, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-input-150.bz2): Stopped
  60. Sep 6 14:53:35 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  61. Sep 6 14:53:35 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  62. Sep 6 14:53:35 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  63. Sep 6 14:53:35 node2 pengine[1712]: notice: Promote ClusterDB:0#011(Slave -> Master node2)
  64. Sep 6 14:53:35 node2 pengine[1712]: notice: Start WebSite#011(node2)
  65. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 32: start WebSite_start_0 on node2 (local)
  66. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 42: notify ClusterDB_pre_notify_promote_0 on node2 (local)
  67. Sep 6 14:53:35 node2 pengine[1712]: notice: Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-151.bz2
  68. Sep 6 14:53:35 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=177, rc=0, cib-update=0, confirmed=true)
  69. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 7: promote ClusterDB_promote_0 on node2 (local)
  70. Sep 6 14:53:36 node2 apache(WebSite)[6956]: INFO: httpd: Could not reliably determine the server's fully qualified domain name, using node2.domain for ServerName
  71. Sep 6 14:53:36 node2 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  72. Sep 6 14:53:36 node2 crmd[1713]: notice: Operation WebSite_start_0: ok (node=node2, call=176, rc=0, cib-update=254, confirmed=true)
  73. Sep 6 14:53:36 node2 crmd[1713]: notice: Initiating action 33: monitor WebSite_monitor_60000 on node2 (local)
  74. Sep 6 14:53:37 node2 crm-fence-peer.sh[7109]: invoked for clusterdb
  75. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition aborted by rsc_location.drbd-fence-by-handler-clusterdb-ClusterDBclone 'create': Non-status change (cib=0.103.0, source=te_update_diff:436, path=/cib/configuration/constraints, 0)
  76. Sep 6 14:53:37 node2 crm-fence-peer.sh[7109]: INFO peer is not reachable, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-clusterdb-ClusterDBclone'
  77. Sep 6 14:53:37 node2 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 5 (0x500)
  78. Sep 6 14:53:37 node2 kernel: block drbd0: fence-peer helper returned 5 (peer is unreachable, assumed to be dead)
  79. Sep 6 14:53:37 node2 kernel: block drbd0: role( Secondary -> Primary ) pdsk( DUnknown -> Outdated )
  80. Sep 6 14:53:37 node2 kernel: block drbd0: new current UUID 08E87D6F755A6305:8B8DEE9DDC86DE72:40688BF57686889E:40678BF57686889F
  81. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_promote_0: ok (node=node2, call=178, rc=0, cib-update=256, confirmed=true)
  82. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 43: notify ClusterDB_post_notify_promote_0 on node2 (local)
  83. Sep 6 14:53:37 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (10000)
  84. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=180, rc=0, cib-update=0, confirmed=true)
  85. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition 43 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-151.bz2): Complete
  86. Sep 6 14:53:37 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  87. Sep 6 14:53:37 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  88. Sep 6 14:53:37 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  89. Sep 6 14:53:37 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Slave node2)
  90. Sep 6 14:53:37 node2 attrd[1711]: notice: Sent update 203: master-ClusterDB=10000
  91. Sep 6 14:53:37 node2 pengine[1712]: notice: Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-152.bz2
  92. Sep 6 14:53:37 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  93. Sep 6 14:53:37 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  94. Sep 6 14:53:37 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  95. Sep 6 14:53:37 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Slave node2)
  96. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 45: notify ClusterDB_pre_notify_demote_0 on node2 (local)
  97. Sep 6 14:53:37 node2 pengine[1712]: notice: Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-153.bz2
  98. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=181, rc=0, cib-update=0, confirmed=true)
  99. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 7: demote ClusterDB_demote_0 on node2 (local)
  100. Sep 6 14:53:37 node2 kernel: block drbd0: role( Primary -> Secondary )
  101. Sep 6 14:53:37 node2 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  102. Sep 6 14:53:37 node2 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  103. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_demote_0: ok (node=node2, call=182, rc=0, cib-update=259, confirmed=true)
  104. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 46: notify ClusterDB_post_notify_demote_0 on node2 (local)
  105. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=183, rc=0, cib-update=0, confirmed=true)
  106. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 9: monitor ClusterDB_monitor_60000 on node2 (local)
  107. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition 45 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-153.bz2): Complete
  108. Sep 6 14:53:37 node2 crmd[1713]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!