Advertisement
Guest User

PacemakerDRBDfailover

a guest
Sep 6th, 2016
131
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Sep 6 14:50:18 node2 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  2. Sep 6 14:51:18 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (10000)
  3. Sep 6 14:51:18 node2 attrd[1711]: notice: Sent update 197: master-ClusterDB=10000
  4. Sep 6 14:51:18 node2 crmd[1713]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  5. Sep 6 14:51:18 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  6. Sep 6 14:51:18 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  7. Sep 6 14:51:18 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  8. Sep 6 14:51:18 node2 crmd[1713]: notice: Transition 41 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-149.bz2): Complete
  9. Sep 6 14:51:18 node2 crmd[1713]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  10. Sep 6 14:51:18 node2 pengine[1712]: notice: Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-149.bz2
  11. Sep 6 14:53:33 node2 crmd[1713]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  12. Sep 6 14:53:33 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  13. Sep 6 14:53:33 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  14. Sep 6 14:53:33 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  15. Sep 6 14:53:33 node2 pengine[1712]: notice: Move Cluster_VIP#011(Started node1 -> node2)
  16. Sep 6 14:53:33 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Stopped node1)
  17. Sep 6 14:53:33 node2 pengine[1712]: notice: Promote ClusterDB:1#011(Slave -> Master node2)
  18. Sep 6 14:53:33 node2 pengine[1712]: notice: Stop ClusterFS#011(node1)
  19. Sep 6 14:53:33 node2 pengine[1712]: notice: Move WebSite#011(Started node1 -> node2)
  20. Sep 6 14:53:33 node2 pengine[1712]: notice: Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-150.bz2
  21. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 6: stop Cluster_VIP_stop_0 on node1
  22. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 4: cancel ClusterDB_monitor_60000 on node2 (local)
  23. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 38: stop ClusterFS_stop_0 on node1
  24. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 54: notify ClusterDB_pre_notify_demote_0 on node2 (local)
  25. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=170, rc=0, cib-update=0, confirmed=true)
  26. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 7: start Cluster_VIP_start_0 on node2 (local)
  27. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 52: notify ClusterDB_pre_notify_demote_0 on node1
  28. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: Adding inet address 10.0.2.200/24 with broadcast address 10.0.2.255 to device eth2
  29. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: Bringing device eth2 up
  30. Sep 6 14:53:33 node2 IPaddr2(Cluster_VIP)[6745]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.2.200 eth2 10.0.2.200 auto not_used not_used
  31. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation Cluster_VIP_start_0: ok (node=node2, call=171, rc=0, cib-update=251, confirmed=true)
  32. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 8: monitor Cluster_VIP_monitor_10000 on node2 (local)
  33. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 39: stop WebSite_stop_0 on node1
  34. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 9: demote ClusterDB_demote_0 on node1
  35. Sep 6 14:53:33 node2 kernel: block drbd0: peer( Primary -> Secondary )
  36. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 53: notify ClusterDB_post_notify_demote_0 on node1
  37. Sep 6 14:53:33 node2 crmd[1713]: notice: Initiating action 55: notify ClusterDB_post_notify_demote_0 on node2 (local)
  38. Sep 6 14:53:33 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=173, rc=0, cib-update=0, confirmed=true)
  39. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 47: notify ClusterDB_pre_notify_stop_0 on node1
  40. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 48: notify ClusterDB_pre_notify_stop_0 on node2 (local)
  41. Sep 6 14:53:34 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=174, rc=0, cib-update=0, confirmed=true)
  42. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 10: stop ClusterDB_stop_0 on node1
  43. Sep 6 14:53:34 node2 kernel: block drbd0: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
  44. Sep 6 14:53:34 node2 kernel: block drbd0: meta connection shut down by peer.
  45. Sep 6 14:53:34 node2 kernel: block drbd0: asender terminated
  46. Sep 6 14:53:34 node2 kernel: block drbd0: Terminating drbd0_asender
  47. Sep 6 14:53:34 node2 kernel: block drbd0: Connection closed
  48. Sep 6 14:53:34 node2 kernel: block drbd0: conn( TearDown -> Unconnected )
  49. Sep 6 14:53:34 node2 kernel: block drbd0: receiver terminated
  50. Sep 6 14:53:34 node2 kernel: block drbd0: Restarting drbd0_receiver
  51. Sep 6 14:53:34 node2 kernel: block drbd0: receiver (re)started
  52. Sep 6 14:53:34 node2 kernel: block drbd0: conn( Unconnected -> WFConnection )
  53. Sep 6 14:53:34 node2 crmd[1713]: notice: Initiating action 49: notify ClusterDB_post_notify_stop_0 on node2 (local)
  54. Sep 6 14:53:34 node2 crmd[1713]: warning: No match for shutdown action on node1
  55. Sep 6 14:53:34 node2 crmd[1713]: notice: Transition aborted by deletion of nvpair[@id='status-node1-master-ClusterDB']: Transient attribute change (cib=0.102.8, source=abort_unless_down:343, path=/cib/status/node_state[@id='node1']/transient_attributes[@id='node1']/instance_attributes[@id='status-node1']/nvpair[@id='status-node1-master-ClusterDB'], 0)
  56. Sep 6 14:53:34 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (1000)
  57. Sep 6 14:53:34 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=175, rc=0, cib-update=0, confirmed=true)
  58. Sep 6 14:53:34 node2 attrd[1711]: notice: Sent update 201: master-ClusterDB=1000
  59. Sep 6 14:53:35 node2 crmd[1713]: notice: Transition 42 (Complete=29, Pending=0, Fired=0, Skipped=2, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-input-150.bz2): Stopped
  60. Sep 6 14:53:35 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  61. Sep 6 14:53:35 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  62. Sep 6 14:53:35 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  63. Sep 6 14:53:35 node2 pengine[1712]: notice: Promote ClusterDB:0#011(Slave -> Master node2)
  64. Sep 6 14:53:35 node2 pengine[1712]: notice: Start WebSite#011(node2)
  65. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 32: start WebSite_start_0 on node2 (local)
  66. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 42: notify ClusterDB_pre_notify_promote_0 on node2 (local)
  67. Sep 6 14:53:35 node2 pengine[1712]: notice: Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-151.bz2
  68. Sep 6 14:53:35 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=177, rc=0, cib-update=0, confirmed=true)
  69. Sep 6 14:53:35 node2 crmd[1713]: notice: Initiating action 7: promote ClusterDB_promote_0 on node2 (local)
  70. Sep 6 14:53:36 node2 apache(WebSite)[6956]: INFO: httpd: Could not reliably determine the server's fully qualified domain name, using node2.domain for ServerName
  71. Sep 6 14:53:36 node2 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0
  72. Sep 6 14:53:36 node2 crmd[1713]: notice: Operation WebSite_start_0: ok (node=node2, call=176, rc=0, cib-update=254, confirmed=true)
  73. Sep 6 14:53:36 node2 crmd[1713]: notice: Initiating action 33: monitor WebSite_monitor_60000 on node2 (local)
  74. Sep 6 14:53:37 node2 crm-fence-peer.sh[7109]: invoked for clusterdb
  75. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition aborted by rsc_location.drbd-fence-by-handler-clusterdb-ClusterDBclone 'create': Non-status change (cib=0.103.0, source=te_update_diff:436, path=/cib/configuration/constraints, 0)
  76. Sep 6 14:53:37 node2 crm-fence-peer.sh[7109]: INFO peer is not reachable, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-clusterdb-ClusterDBclone'
  77. Sep 6 14:53:37 node2 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 5 (0x500)
  78. Sep 6 14:53:37 node2 kernel: block drbd0: fence-peer helper returned 5 (peer is unreachable, assumed to be dead)
  79. Sep 6 14:53:37 node2 kernel: block drbd0: role( Secondary -> Primary ) pdsk( DUnknown -> Outdated )
  80. Sep 6 14:53:37 node2 kernel: block drbd0: new current UUID 08E87D6F755A6305:8B8DEE9DDC86DE72:40688BF57686889E:40678BF57686889F
  81. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_promote_0: ok (node=node2, call=178, rc=0, cib-update=256, confirmed=true)
  82. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 43: notify ClusterDB_post_notify_promote_0 on node2 (local)
  83. Sep 6 14:53:37 node2 attrd[1711]: notice: Sending flush op to all hosts for: master-ClusterDB (10000)
  84. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=180, rc=0, cib-update=0, confirmed=true)
  85. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition 43 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-151.bz2): Complete
  86. Sep 6 14:53:37 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  87. Sep 6 14:53:37 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  88. Sep 6 14:53:37 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  89. Sep 6 14:53:37 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Slave node2)
  90. Sep 6 14:53:37 node2 attrd[1711]: notice: Sent update 203: master-ClusterDB=10000
  91. Sep 6 14:53:37 node2 pengine[1712]: notice: Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-152.bz2
  92. Sep 6 14:53:37 node2 pengine[1712]: notice: On loss of CCM Quorum: Ignore
  93. Sep 6 14:53:37 node2 pengine[1712]: warning: Processing failed op start for ClusterFS on node2: unknown error (1)
  94. Sep 6 14:53:37 node2 pengine[1712]: warning: Forcing ClusterFS away from node2 after 1000000 failures (max=1000000)
  95. Sep 6 14:53:37 node2 pengine[1712]: notice: Demote ClusterDB:0#011(Master -> Slave node2)
  96. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 45: notify ClusterDB_pre_notify_demote_0 on node2 (local)
  97. Sep 6 14:53:37 node2 pengine[1712]: notice: Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-153.bz2
  98. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=181, rc=0, cib-update=0, confirmed=true)
  99. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 7: demote ClusterDB_demote_0 on node2 (local)
  100. Sep 6 14:53:37 node2 kernel: block drbd0: role( Primary -> Secondary )
  101. Sep 6 14:53:37 node2 kernel: block drbd0: bitmap WRITE of 0 pages took 0 jiffies
  102. Sep 6 14:53:37 node2 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  103. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_demote_0: ok (node=node2, call=182, rc=0, cib-update=259, confirmed=true)
  104. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 46: notify ClusterDB_post_notify_demote_0 on node2 (local)
  105. Sep 6 14:53:37 node2 crmd[1713]: notice: Operation ClusterDB_notify_0: ok (node=node2, call=183, rc=0, cib-update=0, confirmed=true)
  106. Sep 6 14:53:37 node2 crmd[1713]: notice: Initiating action 9: monitor ClusterDB_monitor_60000 on node2 (local)
  107. Sep 6 14:53:37 node2 crmd[1713]: notice: Transition 45 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-153.bz2): Complete
  108. Sep 6 14:53:37 node2 crmd[1713]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Advertisement
RAW Paste Data Copied
Advertisement