Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

killwsguardian1

By: a guest on Feb 6th, 2012  |  syntax: None  |  size: 6.58 KB  |  views: 81  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. In node1
  2.  
  3. Feb  6 09:59:42 wsguardian1 corosync[2725]: cman killed by node 2 because we were killed by cman_tool or other application
  4. Feb  6 09:59:42 wsguardian1 fenced[2784]: cluster is down, exiting
  5. Feb  6 09:59:42 wsguardian1 gfs_controld[2859]: cluster is down, exiting
  6. Feb  6 09:59:42 wsguardian1 fenced[2784]: daemon cpg_dispatch error 2
  7. Feb  6 09:59:42 wsguardian1 gfs_controld[2859]: daemon cpg_dispatch error 2
  8. Feb  6 09:59:42 wsguardian1 dlm_controld[2800]: cluster is down, exiting
  9. Feb  6 09:59:42 wsguardian1 fenced[2784]: cpg_dispatch error 2
  10. Feb  6 09:59:42 wsguardian1 dlm_controld[2800]: daemon cpg_dispatch error 2
  11. Feb  6 09:59:44 wsguardian1 kernel: dlm: closing connection to node 2
  12. Feb  6 09:59:44 wsguardian1 kernel: dlm: closing connection to node 1
  13.  
  14.  
  15.  
  16. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: PingAck did not arrive in time.
  17. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  18. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: asender terminated
  19. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: Terminating asender thread
  20. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: Connection closed
  21. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: conn( NetworkFailure -> Unconnected )
  22. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: receiver terminated
  23. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: Restarting receiver thread
  24. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: receiver (re)started
  25. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: conn( Unconnected -> WFConnection )
  26. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: helper command: /sbin/drbdadm fence-peer wsg_db
  27. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: PingAck did not arrive in time.
  28. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )
  29. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: asender terminated
  30. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: Terminating asender thread
  31. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: Connection closed
  32. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: conn( NetworkFailure -> Unconnected )
  33. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: receiver terminated
  34. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: Restarting receiver thread
  35. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: receiver (re)started
  36. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: conn( Unconnected -> WFConnection )
  37. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: helper command: /sbin/drbdadm fence-peer wsg_config
  38. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  39. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  40. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  41. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  42. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  43. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  44. Feb  6 10:00:03 wsguardian1 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  45. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_db]
  46. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [0]
  47. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian2]
  48. Feb  6 10:00:03 wsguardian1 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  49. Feb  6 10:00:03 wsguardian1 rhcs_fence: 460; DEBUG: output: /usr/sbin/cman_tool: Cannot open connection to cman, is it running ?
  50. Feb  6 10:00:03 wsguardian1 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [256]
  51. Feb  6 10:00:03 wsguardian1 rhcs_fence: 471; DEBUG: I am: []
  52. Feb  6 10:00:03 wsguardian1 rhcs_fence: 474; Unable to find local node name.
  53. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: helper command: /sbin/drbdadm fence-peer wsg_db exit code 1 (0x100)
  54. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_db: fence-peer helper broken, returned 1
  55. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [uname] is at: [/bin/uname]
  56. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  57. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [fence_node] is at: [/usr/sbin/fence_node]
  58. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  59. Feb  6 10:00:03 wsguardian1 rhcs_fence: 125; DEBUG: Checking if: [cman_tool] is at: [/usr/sbin/cman_tool]
  60. Feb  6 10:00:03 wsguardian1 rhcs_fence: 156; DEBUG: Found!
  61. Feb  6 10:00:03 wsguardian1 rhcs_fence: 74; Attempting to fence peer using RHCS from DRBD...
  62. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_RESOURCE] == [wsg_config]
  63. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_MINOR] == [1]
  64. Feb  6 10:00:03 wsguardian1 rhcs_fence: 80; DEBUG: Environment variable: [DRBD_PEERS] == [wsguardian2]
  65. Feb  6 10:00:03 wsguardian1 rhcs_fence: 454; DEBUG: shell call: [/usr/sbin/cman_tool status]
  66. Feb  6 10:00:03 wsguardian1 rhcs_fence: 460; DEBUG: output: /usr/sbin/cman_tool: Cannot open connection to cman, is it running ?
  67. Feb  6 10:00:03 wsguardian1 rhcs_fence: 469; DEBUG: Attempt to get local node name via 'cman_tool status' exited with: [256]
  68. Feb  6 10:00:03 wsguardian1 rhcs_fence: 471; DEBUG: I am: []
  69. Feb  6 10:00:03 wsguardian1 rhcs_fence: 474; Unable to find local node name.
  70. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: helper command: /sbin/drbdadm fence-peer wsg_config exit code 1 (0x100)
  71. Feb  6 10:00:03 wsguardian1 kernel: d-con wsg_config: fence-peer helper broken, returned 1
  72.  
  73.  
  74.  
  75. In node 2
  76.  
  77. Feb  6 09:59:52 wsguardian2 corosync[2668]:   [TOTEM ] A processor failed, forming new configuration.
  78. Feb  6 09:59:54 wsguardian2 corosync[2668]:   [QUORUM] Members[1]: 2
  79. Feb  6 09:59:54 wsguardian2 corosync[2668]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
  80. Feb  6 09:59:54 wsguardian2 corosync[2668]:   [CPG   ] chosen downlist: sender r(0) ip(192.168.253.2) ; members(old:2 left:1)
  81. Feb  6 09:59:54 wsguardian2 corosync[2668]:   [MAIN  ] Completed service synchronization, ready to provide service.
  82. Feb  6 09:59:54 wsguardian2 kernel: dlm: closing connection to node 1
  83. Feb  6 09:59:54 wsguardian2 fenced[2727]: fencing node wsguardian1
  84.  
  85.  
  86. The node 2 halt complety after i execute the command /usr/sbin/cman_tool kill -n wsguardian1 from this