API tools faq
paste
Advertisement
Guest User

tail -f /var/log/syslog

a guest
Aug 10th, 2013
102
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 32.98 KB | None | 0 0
raw download clone embed print report
  1. Aug 10 21:32:54 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  2. Aug 10 21:32:54 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1065 (ref=pe_calc-dc-1376163174-3531) derived from /var/lib/pengine/pe-input-1579.bz2
  3. Aug 10 21:32:54 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 118: monitor p_drbd_nfs:0_monitor_20000 on ha2
  4. Aug 10 21:32:54 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 123: monitor p_drbd_nfs:1_monitor_20000 on ha1 (local)
  5. Aug 10 21:32:54 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 monitor[881] (pid 19996)
  6. Aug 10 21:32:54 ha1 pengine: [2721]: notice: process_pe_message: Transition 1065: PEngine Input stored in: /var/lib/pengine/pe-input-1579.bz2
  7. Aug 10 21:32:54 ha1 lrmd: [2716]: info: operation monitor[881] on p_drbd_nfs:1 for client 2722: pid 19996 exited with return code 8
  8. Aug 10 21:32:54 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_monitor_20000 (call=881, rc=8, cib-update=2451, confirmed=false) master
  9. Aug 10 21:32:54 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1065 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1579.bz2): Complete
  10. Aug 10 21:32:54 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  11. Aug 10 21:33:19 ha1 cib: [2710]: info: cib_process_request: Operation complete: op cib_delete for section constraints (origin=ha2/crm_resource/3, version=0.660.37): ok (rc=0)
  12. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <cib admin_epoch="0" epoch="660" num_updates="37" >
  13. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <configuration >
  14. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <constraints >
  15. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <rsc_location id="cli-prefer-nfs" >
  16. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <rule id="cli-prefer-rule-nfs" >
  17. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - <expression value="ha1" id="cli-prefer-expr-nfs" />
  18. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - </rule>
  19. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - </rsc_location>
  20. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - </constraints>
  21. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - </configuration>
  22. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: - </cib>
  23. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <cib epoch="661" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="ha2" update-client="crmd" cib-last-written="Sat Aug 10 21:32:53 2013" have-quorum="1" dc-uuid="ha1" >
  24. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <configuration >
  25. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <constraints >
  26. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <rsc_location id="cli-prefer-nfs" rsc="nfs" >
  27. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <rule id="cli-prefer-rule-nfs" score="INFINITY" boolean-op="and" >
  28. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + <expression id="cli-prefer-expr-nfs" attribute="#uname" operation="eq" value="ha2" type="string" />
  29. Aug 10 21:33:19 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.661.1) : Non-status change
  30. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + </rule>
  31. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + </rsc_location>
  32. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + </constraints>
  33. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + </configuration>
  34. Aug 10 21:33:19 ha1 cib: [2710]: info: cib:diff: + </cib>
  35. Aug 10 21:33:19 ha1 cib: [2710]: info: cib_process_request: Operation complete: op cib_modify for section constraints (origin=ha2/crm_resource/4, version=0.661.1): ok (rc=0)
  36. Aug 10 21:33:19 ha1 crmd: [2722]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  37. Aug 10 21:33:19 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  38. Aug 10 21:33:19 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:1 active in master mode on ha1
  39. Aug 10 21:33:19 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  40. Aug 10 21:33:19 ha1 pengine: [2721]: notice: LogActions: Migrate nfs#011(Started ha1 -> ha2)
  41. Aug 10 21:33:19 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  42. Aug 10 21:33:19 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1066 (ref=pe_calc-dc-1376163199-3534) derived from /var/lib/pengine/pe-input-1580.bz2
  43. Aug 10 21:33:19 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 244: migrate_to nfs_migrate_to_0 on ha1 (local)
  44. Aug 10 21:33:19 ha1 lrmd: [2716]: info: cancel_op: operation monitor[871] on nfs for client 2722, its parameters: crm_feature_set=[3.0.6] xmfile=[/cluster/xen/nfs.cfg] CRM_meta_name=[monitor] CRM_meta_interval=[10000] CRM_meta_timeout=[20000] cancelled
  45. Aug 10 21:33:19 ha1 lrmd: [2716]: info: rsc:nfs migrate_to[882] (pid 20175)
  46. Aug 10 21:33:19 ha1 crmd: [2722]: info: process_lrm_event: LRM operation nfs_monitor_10000 (call=871, status=1, cib-update=0, confirmed=true) Cancelled
  47. Aug 10 21:33:19 ha1 pengine: [2721]: notice: process_pe_message: Transition 1066: PEngine Input stored in: /var/lib/pengine/pe-input-1580.bz2
  48. Aug 10 21:33:20 ha1 Xen[20175]: INFO: nfs: Starting xm migrate to ha2
  49. Aug 10 21:33:24 ha1 kernel: [526517.250602] xenbr0: port 2(vif48.0) entering forwarding state
  50. Aug 10 21:33:24 ha1 kernel: [526517.250838] device vif48.0 left promiscuous mode
  51. Aug 10 21:33:24 ha1 kernel: [526517.250848] xenbr0: port 2(vif48.0) entering disabled state
  52. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/console/48/0
  53. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/48/51713
  54. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/vif-bridge: offline type_if=vif XENBUS_PATH=backend/vif/48/0
  55. Aug 10 21:33:24 ha1 Xen[20175]: INFO: nfs: xm migrate to ha2 succeeded.
  56. Aug 10 21:33:24 ha1 lrmd: [2716]: info: operation migrate_to[882] on nfs for client 2722: pid 20175 exited with return code 0
  57. Aug 10 21:33:24 ha1 crmd: [2722]: info: process_lrm_event: LRM operation nfs_migrate_to_0 (call=882, rc=0, cib-update=2453, confirmed=true) ok
  58. Aug 10 21:33:24 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 245: migrate_from nfs_migrate_from_0 on ha2
  59. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/block-drbd: remove XENBUS_PATH=backend/vbd/48/51713
  60. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/vif-bridge: brctl delif xenbr0 vif48.0 failed
  61. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/vif-bridge: ifconfig vif48.0 down failed
  62. Aug 10 21:33:24 ha1 kernel: [526517.668034] block drbd4: role( Primary -> Secondary )
  63. Aug 10 21:33:24 ha1 kernel: [526517.668222] block drbd4: bitmap WRITE of 0 pages took 0 jiffies
  64. Aug 10 21:33:24 ha1 kernel: [526517.682183] block drbd4: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  65. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge offline for vif48.0, bridge xenbr0.
  66. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/48/51713
  67. Aug 10 21:33:24 ha1 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vif/48/0
  68. Aug 10 21:33:25 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 10: stop nfs_stop_0 on ha1 (local)
  69. Aug 10 21:33:25 ha1 lrmd: [2716]: info: rsc:nfs stop[883] (pid 20418)
  70. Aug 10 21:33:26 ha1 ntpd[2443]: Deleting interface #58 vif48.0, fe80::fcff:ffff:feff:ffff#123, interface stats: received=0, sent=0, dropped=0, active_time=143 secs
  71. Aug 10 21:33:26 ha1 ntpd[2443]: peers refreshed
  72. Aug 10 21:33:27 ha1 Xen[20418]: INFO: Xen domain nfs already stopped.
  73. Aug 10 21:33:27 ha1 lrmd: [2716]: info: RA output: (nfs:stop:stderr) Error: Domain 'nfs' does not exist.
  74. Aug 10 21:33:27 ha1 lrmd: [2716]: info: operation stop[883] on nfs for client 2722: pid 20418 exited with return code 0
  75. Aug 10 21:33:27 ha1 crmd: [2722]: info: process_lrm_event: LRM operation nfs_stop_0 (call=883, rc=0, cib-update=2454, confirmed=true) ok
  76. Aug 10 21:33:27 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 12: monitor nfs_monitor_10000 on ha2
  77. Aug 10 21:33:28 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1066 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1580.bz2): Complete
  78. Aug 10 21:33:28 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  79. Aug 10 21:33:35 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_monitor_20000 (call=881, rc=0, cib-update=2455, confirmed=false) ok
  80. Aug 10 21:33:35 ha1 crmd: [2722]: info: process_graph_event: Detected action p_drbd_nfs:1_monitor_20000 from a different transition: 1065 vs. 1066
  81. Aug 10 21:33:35 ha1 crmd: [2722]: info: abort_transition_graph: process_graph_event:476 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=p_drbd_nfs:1_last_failure_0, magic=0:0;123:1065:8:60a16b49-0004-4ae1-a0f5-b87b3a38f48b, cib=0.661.6) : Old event
  82. Aug 10 21:33:35 ha1 crmd: [2722]: WARN: update_failcount: Updating failcount for p_drbd_nfs:1 on ha1 after failed monitor: rc=0 (update=value++, time=1376163215)
  83. Aug 10 21:33:35 ha1 attrd: [2718]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_drbd_nfs:1 (1)
  84. Aug 10 21:33:35 ha1 crmd: [2722]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
  85. Aug 10 21:33:35 ha1 attrd: [2718]: notice: attrd_perform_update: Sent update 2982: fail-count-p_drbd_nfs:1=1
  86. Aug 10 21:33:35 ha1 attrd: [2718]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_drbd_nfs:1 (1376163215)
  87. Aug 10 21:33:35 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-ha1-fail-count-p_drbd_nfs.1, name=fail-count-p_drbd_nfs:1, value=1, magic=NA, cib=0.661.7) : Transient attribute: update
  88. Aug 10 21:33:35 ha1 attrd: [2718]: notice: attrd_perform_update: Sent update 2984: last-failure-p_drbd_nfs:1=1376163215
  89. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  90. Aug 10 21:33:35 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  91. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  92. Aug 10 21:33:35 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-ha1-last-failure-p_drbd_nfs.1, name=last-failure-p_drbd_nfs:1, value=1376163215, magic=NA, cib=0.661.8) : Transient attribute: update
  93. Aug 10 21:33:35 ha1 pengine: [2721]: notice: LogActions: Recover p_drbd_nfs:1#011(Master ha1)
  94. Aug 10 21:33:35 ha1 crmd: [2722]: info: handle_response: pe_calc calculation pe_calc-dc-1376163215-3539 is obsolete
  95. Aug 10 21:33:35 ha1 pengine: [2721]: notice: process_pe_message: Transition 1067: PEngine Input stored in: /var/lib/pengine/pe-input-1581.bz2
  96. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  97. Aug 10 21:33:35 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  98. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  99. Aug 10 21:33:35 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  100. Aug 10 21:33:35 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  101. Aug 10 21:33:35 ha1 pengine: [2721]: notice: LogActions: Recover p_drbd_nfs:1#011(Master ha1)
  102. Aug 10 21:33:35 ha1 crmd: [2722]: info: handle_response: pe_calc calculation pe_calc-dc-1376163215-3540 is obsolete
  103. Aug 10 21:33:35 ha1 pengine: [2721]: notice: process_pe_message: Transition 1068: PEngine Input stored in: /var/lib/pengine/pe-input-1582.bz2
  104. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  105. Aug 10 21:33:35 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  106. Aug 10 21:33:35 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  107. Aug 10 21:33:35 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  108. Aug 10 21:33:35 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  109. Aug 10 21:33:35 ha1 pengine: [2721]: notice: LogActions: Recover p_drbd_nfs:1#011(Master ha1)
  110. Aug 10 21:33:35 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  111. Aug 10 21:33:35 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1069 (ref=pe_calc-dc-1376163215-3541) derived from /var/lib/pengine/pe-input-1583.bz2
  112. Aug 10 21:33:35 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 269: notify p_drbd_nfs:0_pre_notify_demote_0 on ha2
  113. Aug 10 21:33:35 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 271: notify p_drbd_nfs:1_pre_notify_demote_0 on ha1 (local)
  114. Aug 10 21:33:35 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[884] (pid 20587)
  115. Aug 10 21:33:35 ha1 pengine: [2721]: notice: process_pe_message: Transition 1069: PEngine Input stored in: /var/lib/pengine/pe-input-1583.bz2
  116. Aug 10 21:33:35 ha1 lrmd: [2716]: info: operation notify[884] on p_drbd_nfs:1 for client 2722: pid 20587 exited with return code 0
  117. Aug 10 21:33:35 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=884, rc=0, cib-update=0, confirmed=true) ok
  118. Aug 10 21:33:35 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 121: demote p_drbd_nfs:1_demote_0 on ha1 (local)
  119. Aug 10 21:33:35 ha1 lrmd: [2716]: info: cancel_op: operation monitor[881] on p_drbd_nfs:1 for client 2722, its parameters: drbd_resource=[nfs] CRM_meta_role=[Master] CRM_meta_timeout=[60000] CRM_meta_name=[monitor] crm_feature_set=[3.0.6] CRM_meta_notify=[true] CRM_meta_clone_node_max=[1] CRM_meta_clone=[1] CRM_meta_clone_max=[2] CRM_meta_master_node_max=[1] CRM_meta_interval=[20000] CRM_meta_globally_unique=[false] CRM_meta_master_max=[2] cancelled
  120. Aug 10 21:33:35 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 demote[885] (pid 20610)
  121. Aug 10 21:33:36 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_monitor_20000 (call=881, status=1, cib-update=0, confirmed=true) Cancelled
  122. Aug 10 21:33:36 ha1 lrmd: [2716]: info: operation demote[885] on p_drbd_nfs:1 for client 2722: pid 20610 exited with return code 0
  123. Aug 10 21:33:36 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_demote_0 (call=885, rc=0, cib-update=2459, confirmed=true) ok
  124. Aug 10 21:33:36 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 270: notify p_drbd_nfs:0_post_notify_demote_0 on ha2
  125. Aug 10 21:33:36 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 272: notify p_drbd_nfs:1_post_notify_demote_0 on ha1 (local)
  126. Aug 10 21:33:36 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[886] (pid 20634)
  127. Aug 10 21:33:36 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:notify:stdout)
  128. Aug 10 21:33:36 ha1 lrmd: [2716]: info: operation notify[886] on p_drbd_nfs:1 for client 2722: pid 20634 exited with return code 0
  129. Aug 10 21:33:36 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=886, rc=0, cib-update=0, confirmed=true) ok
  130. Aug 10 21:33:36 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 262: notify p_drbd_nfs:0_pre_notify_stop_0 on ha2
  131. Aug 10 21:33:36 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 264: notify p_drbd_nfs:1_pre_notify_stop_0 on ha1 (local)
  132. Aug 10 21:33:36 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[887] (pid 20662)
  133. Aug 10 21:33:36 ha1 lrmd: [2716]: info: operation notify[887] on p_drbd_nfs:1 for client 2722: pid 20662 exited with return code 0
  134. Aug 10 21:33:36 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=887, rc=0, cib-update=0, confirmed=true) ok
  135. Aug 10 21:33:36 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 3: stop p_drbd_nfs:1_stop_0 on ha1 (local)
  136. Aug 10 21:33:36 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 stop[888] (pid 20685)
  137. Aug 10 21:33:36 ha1 kernel: [526529.713201] block drbd4: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
  138. Aug 10 21:33:36 ha1 kernel: [526529.713341] block drbd4: asender terminated
  139. Aug 10 21:33:36 ha1 kernel: [526529.713353] block drbd4: Terminating drbd4_asender
  140. Aug 10 21:33:36 ha1 kernel: [526529.742751] block drbd4: Connection closed
  141. Aug 10 21:33:36 ha1 kernel: [526529.742823] block drbd4: conn( Disconnecting -> StandAlone )
  142. Aug 10 21:33:36 ha1 kernel: [526529.742904] block drbd4: receiver terminated
  143. Aug 10 21:33:36 ha1 kernel: [526529.742911] block drbd4: Terminating drbd4_receiver
  144. Aug 10 21:33:36 ha1 kernel: [526529.743030] block drbd4: disk( UpToDate -> Failed )
  145. Aug 10 21:33:36 ha1 kernel: [526529.743081] block drbd4: Sending state for detaching disk failed
  146. Aug 10 21:33:36 ha1 kernel: [526529.743149] block drbd4: disk( Failed -> Diskless )
  147. Aug 10 21:33:36 ha1 kernel: [526529.743198] block drbd4: drbd_bm_resize called with capacity == 0
  148. Aug 10 21:33:36 ha1 kernel: [526529.743415] block drbd4: worker terminated
  149. Aug 10 21:33:36 ha1 kernel: [526529.743422] block drbd4: Terminating drbd4_worker
  150. Aug 10 21:33:36 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:stop:stdout)
  151. Aug 10 21:33:36 ha1 attrd: [2718]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd_nfs:1 (<null>)
  152. Aug 10 21:33:36 ha1 attrd: [2718]: notice: attrd_perform_update: Sent delete 2986: node=ha1, attr=master-p_drbd_nfs:1, id=<n/a>, set=(null), section=status
  153. Aug 10 21:33:36 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:stop:stdout)
  154. Aug 10 21:33:36 ha1 lrmd: [2716]: info: operation stop[888] on p_drbd_nfs:1 for client 2722: pid 20685 exited with return code 0
  155. Aug 10 21:33:36 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_stop_0 (call=888, rc=0, cib-update=2460, confirmed=true) ok
  156. Aug 10 21:33:36 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:194 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha1, magic=NA, cib=0.661.11) : Transient attribute: removal
  157. Aug 10 21:33:36 ha1 attrd: [2718]: notice: attrd_perform_update: Sent delete -22: node=ha1, attr=master-p_drbd_nfs:1, id=<n/a>, set=(null), section=status
  158. Aug 10 21:33:37 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 263: notify p_drbd_nfs:0_post_notify_stop_0 on ha2
  159. Aug 10 21:33:37 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1069 (Complete=21, Pending=0, Fired=0, Skipped=13, Incomplete=10, Source=/var/lib/pengine/pe-input-1583.bz2): Stopped
  160. Aug 10 21:33:37 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  161. Aug 10 21:33:37 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  162. Aug 10 21:33:37 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  163. Aug 10 21:33:37 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  164. Aug 10 21:33:37 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  165. Aug 10 21:33:37 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  166. Aug 10 21:33:37 ha1 pengine: [2721]: notice: LogActions: Start p_drbd_nfs:1#011(ha1)
  167. Aug 10 21:33:37 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  168. Aug 10 21:33:37 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1070 (ref=pe_calc-dc-1376163217-3554) derived from /var/lib/pengine/pe-input-1584.bz2
  169. Aug 10 21:33:37 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 258: notify p_drbd_nfs:0_pre_notify_start_0 on ha2
  170. Aug 10 21:33:37 ha1 pengine: [2721]: notice: process_pe_message: Transition 1070: PEngine Input stored in: /var/lib/pengine/pe-input-1584.bz2
  171. Aug 10 21:33:37 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 119: start p_drbd_nfs:1_start_0 on ha1 (local)
  172. Aug 10 21:33:37 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 start[889] (pid 20717)
  173. Aug 10 21:33:37 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout) allow-two-primaries;
  174. Aug 10 21:33:37 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout)
  175. Aug 10 21:33:37 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout)
  176. Aug 10 21:33:37 ha1 kernel: [526530.803882] block drbd4: Starting worker thread (from drbdsetup [20762])
  177. Aug 10 21:33:37 ha1 kernel: [526530.804362] block drbd4: disk( Diskless -> Attaching )
  178. Aug 10 21:33:37 ha1 kernel: [526530.805705] block drbd4: Found 4 transactions (21 active extents) in activity log.
  179. Aug 10 21:33:37 ha1 kernel: [526530.805714] block drbd4: Method to ensure write ordering: flush
  180. Aug 10 21:33:37 ha1 kernel: [526530.805725] block drbd4: drbd_bm_resize called with capacity == 65009656
  181. Aug 10 21:33:37 ha1 kernel: [526530.807169] block drbd4: resync bitmap: bits=8126207 words=126972 pages=248
  182. Aug 10 21:33:37 ha1 kernel: [526530.807179] block drbd4: size = 31 GB (32504828 KB)
  183. Aug 10 21:33:37 ha1 kernel: [526530.834342] block drbd4: bitmap READ of 248 pages took 7 jiffies
  184. Aug 10 21:33:37 ha1 kernel: [526530.837273] block drbd4: recounting of set bits took additional 1 jiffies
  185. Aug 10 21:33:37 ha1 kernel: [526530.837282] block drbd4: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  186. Aug 10 21:33:37 ha1 kernel: [526530.837297] block drbd4: disk( Attaching -> UpToDate )
  187. Aug 10 21:33:37 ha1 kernel: [526530.837304] block drbd4: attached to UUIDs 94222F9BB3BB1E26:0000000000000000:BBCC3322C4784787:BBCB3322C4784787
  188. Aug 10 21:33:38 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout)
  189. Aug 10 21:33:38 ha1 kernel: [526530.923853] block drbd4: conn( StandAlone -> Unconnected )
  190. Aug 10 21:33:38 ha1 kernel: [526530.923915] block drbd4: Starting receiver thread (from drbd4_worker [20763])
  191. Aug 10 21:33:38 ha1 kernel: [526530.924160] block drbd4: receiver (re)started
  192. Aug 10 21:33:38 ha1 kernel: [526530.924181] block drbd4: conn( Unconnected -> WFConnection )
  193. Aug 10 21:33:38 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout)
  194. Aug 10 21:33:38 ha1 attrd: [2718]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd_nfs:1 (1000)
  195. Aug 10 21:33:38 ha1 attrd: [2718]: notice: attrd_perform_update: Sent update 2990: master-p_drbd_nfs:1=1000
  196. Aug 10 21:33:38 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:start:stdout)
  197. Aug 10 21:33:38 ha1 lrmd: [2716]: info: operation start[889] on p_drbd_nfs:1 for client 2722: pid 20717 exited with return code 0
  198. Aug 10 21:33:38 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_start_0 (call=889, rc=0, cib-update=2462, confirmed=true) ok
  199. Aug 10 21:33:38 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-ha1-master-p_drbd_nfs.1, name=master-p_drbd_nfs:1, value=1000, magic=NA, cib=0.661.14) : Transient attribute: update
  200. Aug 10 21:33:38 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 259: notify p_drbd_nfs:0_post_notify_start_0 on ha2
  201. Aug 10 21:33:38 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 260: notify p_drbd_nfs:1_post_notify_start_0 on ha1 (local)
  202. Aug 10 21:33:38 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[890] (pid 20792)
  203. Aug 10 21:33:38 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:notify:stdout)
  204. Aug 10 21:33:38 ha1 lrmd: [2716]: info: operation notify[890] on p_drbd_nfs:1 for client 2722: pid 20792 exited with return code 0
  205. Aug 10 21:33:38 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=890, rc=0, cib-update=0, confirmed=true) ok
  206. Aug 10 21:33:38 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1070 (Complete=10, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-1584.bz2): Stopped
  207. Aug 10 21:33:38 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  208. Aug 10 21:33:38 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  209. Aug 10 21:33:38 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  210. Aug 10 21:33:38 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  211. Aug 10 21:33:38 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  212. Aug 10 21:33:38 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  213. Aug 10 21:33:38 ha1 pengine: [2721]: notice: LogActions: Promote p_drbd_nfs:1#011(Slave -> Master ha1)
  214. Aug 10 21:33:38 ha1 kernel: [526531.421999] block drbd4: Handshake successful: Agreed network protocol version 96
  215. Aug 10 21:33:38 ha1 kernel: [526531.422016] block drbd4: conn( WFConnection -> WFReportParams )
  216. Aug 10 21:33:38 ha1 kernel: [526531.422134] block drbd4: Starting asender thread (from drbd4_receiver [20777])
  217. Aug 10 21:33:38 ha1 kernel: [526531.424941] block drbd4: data-integrity-alg: sha1
  218. Aug 10 21:33:38 ha1 kernel: [526531.425157] block drbd4: drbd_sync_handshake:
  219. Aug 10 21:33:38 ha1 kernel: [526531.425170] block drbd4: self 94222F9BB3BB1E26:0000000000000000:BBCC3322C4784787:BBCB3322C4784787 bits:0 flags:0
  220. Aug 10 21:33:38 ha1 kernel: [526531.425184] block drbd4: peer CC8966C497399DDB:94222F9BB3BB1E27:BBCC3322C4784786:BBCB3322C4784787 bits:0 flags:0
  221. Aug 10 21:33:38 ha1 kernel: [526531.425196] block drbd4: uuid_compare()=-1 by rule 50
  222. Aug 10 21:33:38 ha1 kernel: [526531.425218] block drbd4: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )
  223. Aug 10 21:33:38 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  224. Aug 10 21:33:38 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1071 (ref=pe_calc-dc-1376163218-3560) derived from /var/lib/pengine/pe-input-1585.bz2
  225. Aug 10 21:33:38 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 267: notify p_drbd_nfs:0_pre_notify_promote_0 on ha2
  226. Aug 10 21:33:38 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 269: notify p_drbd_nfs:1_pre_notify_promote_0 on ha1 (local)
  227. Aug 10 21:33:38 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[891] (pid 20823)
  228. Aug 10 21:33:38 ha1 kernel: [526531.483843] block drbd4: conn( WFBitMapT -> WFSyncUUID )
  229. Aug 10 21:33:38 ha1 pengine: [2721]: notice: process_pe_message: Transition 1071: PEngine Input stored in: /var/lib/pengine/pe-input-1585.bz2
  230. Aug 10 21:33:38 ha1 kernel: [526531.524843] block drbd4: updated sync uuid 94232F9BB3BB1E26:0000000000000000:BBCC3322C4784787:BBCB3322C4784787
  231. Aug 10 21:33:38 ha1 kernel: [526531.542990] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4
  232. Aug 10 21:33:38 ha1 kernel: [526531.558092] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4 exit code 0 (0x0)
  233. Aug 10 21:33:38 ha1 kernel: [526531.558111] block drbd4: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
  234. Aug 10 21:33:38 ha1 kernel: [526531.558139] block drbd4: Began resync as SyncTarget (will sync 0 KB [0 bits set]).
  235. Aug 10 21:33:38 ha1 kernel: [526531.560816] block drbd4: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
  236. Aug 10 21:33:38 ha1 kernel: [526531.560837] block drbd4: updated UUIDs CC8966C497399DDA:0000000000000000:94232F9BB3BB1E26:94222F9BB3BB1E27
  237. Aug 10 21:33:38 ha1 kernel: [526531.560856] block drbd4: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
  238. Aug 10 21:33:38 ha1 kernel: [526531.584595] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4
  239. Aug 10 21:33:38 ha1 kernel: [526531.598872] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4 exit code 0 (0x0)
  240. Aug 10 21:33:38 ha1 kernel: [526531.604548] block drbd4: bitmap WRITE of 248 pages took 2 jiffies
  241. Aug 10 21:33:38 ha1 kernel: [526531.617608] block drbd4: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
  242. Aug 10 21:33:38 ha1 lrmd: [2716]: info: operation notify[891] on p_drbd_nfs:1 for client 2722: pid 20823 exited with return code 0
  243. Aug 10 21:33:38 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=891, rc=0, cib-update=0, confirmed=true) ok
  244. Aug 10 21:33:38 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 122: promote p_drbd_nfs:1_promote_0 on ha1 (local)
  245. Aug 10 21:33:38 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 promote[892] (pid 20852)
  246. Aug 10 21:33:39 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:promote:stdout) allow-two-primaries;
  247. Aug 10 21:33:39 ha1 kernel: [526531.898085] block drbd4: role( Secondary -> Primary )
  248. Aug 10 21:33:39 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:promote:stdout)
  249. Aug 10 21:33:39 ha1 lrmd: [2716]: info: operation promote[892] on p_drbd_nfs:1 for client 2722: pid 20852 exited with return code 0
  250. Aug 10 21:33:39 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_promote_0 (call=892, rc=0, cib-update=2464, confirmed=true) ok
  251. Aug 10 21:33:39 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 268: notify p_drbd_nfs:0_post_notify_promote_0 on ha2
  252. Aug 10 21:33:39 ha1 crmd: [2722]: info: te_rsc_command: Initiating action 270: notify p_drbd_nfs:1_post_notify_promote_0 on ha1 (local)
  253. Aug 10 21:33:39 ha1 lrmd: [2716]: info: rsc:p_drbd_nfs:1 notify[893] (pid 20883)
  254. Aug 10 21:33:39 ha1 attrd: [2718]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd_nfs:1 (10000)
  255. Aug 10 21:33:39 ha1 attrd: [2718]: notice: attrd_perform_update: Sent update 2992: master-p_drbd_nfs:1=10000
  256. Aug 10 21:33:39 ha1 lrmd: [2716]: info: RA output: (p_drbd_nfs:1:notify:stdout)
  257. Aug 10 21:33:39 ha1 crmd: [2722]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-ha1-master-p_drbd_nfs.1, name=master-p_drbd_nfs:1, value=10000, magic=NA, cib=0.661.18) : Transient attribute: update
  258. Aug 10 21:33:39 ha1 lrmd: [2716]: info: operation notify[893] on p_drbd_nfs:1 for client 2722: pid 20883 exited with return code 0
  259. Aug 10 21:33:39 ha1 crmd: [2722]: info: process_lrm_event: LRM operation p_drbd_nfs:1_notify_0 (call=893, rc=0, cib-update=0, confirmed=true) ok
  260. Aug 10 21:33:39 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1071 (Complete=11, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-1585.bz2): Stopped
  261. Aug 10 21:33:39 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
  262. Aug 10 21:33:39 ha1 pengine: [2721]: notice: unpack_config: On loss of CCM Quorum: Ignore
  263. Aug 10 21:33:39 ha1 pengine: [2721]: WARN: unpack_rsc_op: Processing failed op p_drbd_nfs:1_last_failure_0 on ha1: ok (0)
  264. Aug 10 21:33:39 ha1 pengine: [2721]: notice: unpack_rsc_op: Operation monitor found resource p_drbd_nfs:0 active in master mode on ha2
  265. Aug 10 21:33:39 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  266. Aug 10 21:33:39 ha1 pengine: [2721]: notice: common_apply_stickiness: ms_drbd_nfs can fail 999999 more times on ha1 before being forced off
  267. Aug 10 21:33:39 ha1 crmd: [2722]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
  268. Aug 10 21:33:39 ha1 crmd: [2722]: info: do_te_invoke: Processing graph 1072 (ref=pe_calc-dc-1376163219-3568) derived from /var/lib/pengine/pe-input-1586.bz2
  269. Aug 10 21:33:39 ha1 crmd: [2722]: notice: run_graph: ==== Transition 1072 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1586.bz2): Complete
  270. Aug 10 21:33:39 ha1 crmd: [2722]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
  271. Aug 10 21:33:39 ha1 pengine: [2721]: notice: process_pe_message: Transition 1072: PEngine Input stored in: /var/lib/pengine/pe-input-1586.bz2
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!