API tools faq
paste
Advertisement
Guest User

ocfs2 hang

a guest
Mar 10th, 2010
460
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 19.52 KB | None | 0 0
raw download clone embed print report
  1. ##################### Here node08 fails. Its off until I will not switch it on again. The ocfs2 disk access hangs
  2.  
  3. Mar 4 04:09:51 ams-farm kernel: o2net: connection to node node08 (num 9) at 192.168.1.8:7777 has been idle for 30.0 seconds, shutting it down.
  4. Mar 4 04:09:51 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267672161.475117 now 1267672191.476304 dr 1267672161.475105 adv 1267672161.475118:1267672161.475118 func (a6c57cb2:504) 1267672125.509372:1267672125.509384)
  5. Mar 4 04:09:51 ams-farm kernel: o2net: no longer connected to node node08 (num 9) at 192.168.1.8:7777
  6. Mar 4 04:10:21 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 9 after 30.0 seconds, giving up and returning errors.
  7. Mar 4 04:10:23 ams-farm kernel: (5786,2):o2dlm_eviction_cb:258 o2dlm has evicted node 9 from group B2F5C3291557493B99AE7326AF8B7471
  8. Mar 4 04:10:23 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 9 from group B2F5C3291557493B99AE7326AF8B7471
  9.  
  10. ##################### ocfs2 disk is still hung. Here I restarted node08 and readded it to the cluster.
  11.  
  12. Mar 4 11:14:10 ams-farm su(pam_unix)[31868]: session opened for user root by alberti(uid=237)
  13. Mar 4 11:14:36 ams-farm kernel: o2net: accepted connection from node node08 (num 9) at 192.168.1.8:7777
  14. Mar 4 11:14:40 ams-farm kernel: ocfs2_dlm: Node 9 joins domain B2F5C3291557493B99AE7326AF8B7471
  15. Mar 4 11:14:40 ams-farm kernel: ocfs2_dlm: Nodes in domain ("B2F5C3291557493B99AE7326AF8B7471"): 0 1 3 4 5 6 7 8 9 10 11
  16.  
  17. ##################### ocfs2 disk is still hung, so I decided to shutdown NOT gracefully all the nodes except one
  18.  
  19. Mar 4 11:20:02 ams-farm kernel: o2net: connection to node node08 (num 9) at 192.168.1.8:7777 has been idle for 30.0 seconds, shutting it down.
  20. Mar 4 11:20:02 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267697972.22362 now 1267698002.23712 dr 1267697972.22347 adv 1267697972.22362:1267697972.22362 func (a6c57cb2:503) 1267697680.163164:1267697680.163179)
  21. Mar 4 11:20:02 ams-farm kernel: o2net: no longer connected to node node08 (num 9) at 192.168.1.8:7777
  22. Mar 4 11:20:19 ams-farm kernel: o2net: connection to node node02 (num 4) at 192.168.1.2:7777 has been idle for 30.0 seconds, shutting it down.
  23. Mar 4 11:20:19 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267697989.971237 now 1267698019.972592 dr 1267697989.971230 adv 1267697989.971238:1267697989.971238 func (a6c57cb2:504) 1267672223.565134:1267672223.565138)
  24. Mar 4 11:20:19 ams-farm kernel: o2net: no longer connected to node node02 (num 4) at 192.168.1.2:7777
  25. Mar 4 11:20:32 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 9 after 30.0 seconds, giving up and returning errors.
  26. Mar 4 11:20:35 ams-farm kernel: o2net: connection to node node03 (num 5) at 192.168.1.3:7777 has been idle for 30.0 seconds, shutting it down.
  27. Mar 4 11:20:35 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698005.853313 now 1267698035.853370 dr 1267698005.853298 adv 1267698005.853320:1267698005.853320 func (a6c57cb2:504) 1267672223.565434:1267672223.565437)
  28. Mar 4 11:20:35 ams-farm kernel: o2net: no longer connected to node node03 (num 5) at 192.168.1.3:7777
  29. Mar 4 11:20:35 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 9 from group B2F5C3291557493B99AE7326AF8B7471
  30. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_get_lock_resource:839 B2F5C3291557493B99AE7326AF8B7471:$RECOVERY: at least one node (9) to recover before lock mastery can begin
  31. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  32. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  33. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  34. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  35. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  36. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 4 went down!
  37. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  38. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 5 went down!
  39. Mar 4 11:20:37 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  40. Mar 4 11:20:49 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 4 after 30.0 seconds, giving up and returning errors.
  41. Mar 4 11:20:51 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 4 from group B2F5C3291557493B99AE7326AF8B7471
  42. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_restart_lock_mastery:1218 ERROR: node down! 4
  43. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_wait_for_lock_mastery:1035 ERROR: status = -11
  44. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  45. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  46. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  47. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 5 went down!
  48. Mar 4 11:20:52 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  49. Mar 4 11:21:05 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 5 after 30.0 seconds, giving up and returning errors.
  50. Mar 4 11:21:09 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 5 from group B2F5C3291557493B99AE7326AF8B7471
  51. Mar 4 11:21:12 ams-farm kernel: (5826,0):dlm_restart_lock_mastery:1218 ERROR: node down! 5
  52. Mar 4 11:21:12 ams-farm kernel: (5826,0):dlm_wait_for_lock_mastery:1035 ERROR: status = -11
  53. Mar 4 11:21:12 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  54. Mar 4 11:21:12 ams-farm kernel: (5826,0):dlm_do_recovery:524 (5826) Node 1 is the Recovery Master for the Dead Node 9 for Domain B2F5C3291557493B99AE7326AF8B7471
  55. Mar 4 11:21:17 ams-farm kernel: (5826,0):dlm_do_recovery:524 (5826) Node 1 is the Recovery Master for the Dead Node 4 for Domain B2F5C3291557493B99AE7326AF8B7471
  56. Mar 4 11:21:22 ams-farm kernel: (5826,0):dlm_do_recovery:524 (5826) Node 1 is the Recovery Master for the Dead Node 5 for Domain B2F5C3291557493B99AE7326AF8B7471
  57. Mar 4 11:23:31 ams-farm kernel: o2net: connection to node node04 (num 6) at 192.168.1.4:7777 has been idle for 30.0 seconds, shutting it down.
  58. Mar 4 11:23:31 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698181.379053 now 1267698211.377975 dr 1267698181.379032 adv 1267698181.379062:1267698181.379063 func (a6c57cb2:507) 1267698084.72878:1267698084.72882)
  59. Mar 4 11:23:31 ams-farm kernel: o2net: no longer connected to node node04 (num 6) at 192.168.1.4:7777
  60. Mar 4 11:23:37 ams-farm kernel: o2net: connection to node node05 (num 7) at 192.168.1.5:7777 has been idle for 30.0 seconds, shutting it down.
  61. Mar 4 11:23:37 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698187.384901 now 1267698217.386271 dr 1267698187.384894 adv 1267698187.384901:1267698187.384902 func (a6c57cb2:516) 1267698082.448055:1267698082.448056)
  62. Mar 4 11:23:37 ams-farm kernel: o2net: no longer connected to node node05 (num 7) at 192.168.1.5:7777
  63. Mar 4 11:23:43 ams-farm kernel: o2net: connection to node node06 (num 8) at 192.168.1.6:7777 has been idle for 30.0 seconds, shutting it down.
  64. Mar 4 11:23:43 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698193.384470 now 1267698223.386565 dr 1267698193.384462 adv 1267698193.384477:1267698193.384477 func (a6c57cb2:507) 1267698083.463042:1267698083.463046)
  65. Mar 4 11:23:43 ams-farm kernel: o2net: no longer connected to node node06 (num 8) at 192.168.1.6:7777
  66. Mar 4 11:24:01 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 6 after 30.0 seconds, giving up and returning errors.
  67. Mar 4 11:24:04 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 6 from group B2F5C3291557493B99AE7326AF8B7471
  68. Mar 4 11:24:07 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 7 after 30.0 seconds, giving up and returning errors.
  69. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_get_lock_resource:839 B2F5C3291557493B99AE7326AF8B7471:$RECOVERY: at least one node (6) to recover before lock mastery can begin
  70. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  71. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  72. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  73. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  74. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  75. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 7 went down!
  76. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  77. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 8 went down!
  78. Mar 4 11:24:07 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  79. Mar 4 11:24:12 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 7 from group B2F5C3291557493B99AE7326AF8B7471
  80. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_restart_lock_mastery:1218 ERROR: node down! 7
  81. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_wait_for_lock_mastery:1035 ERROR: status = -11
  82. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  83. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -107
  84. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -107
  85. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 8 went down!
  86. Mar 4 11:24:12 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  87. Mar 4 11:24:13 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 8 after 30.0 seconds, giving up and returning errors.
  88. Mar 4 11:24:16 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 8 from group B2F5C3291557493B99AE7326AF8B7471
  89. Mar 4 11:24:17 ams-farm kernel: (5826,0):dlm_restart_lock_mastery:1218 ERROR: node down! 8
  90. Mar 4 11:24:17 ams-farm kernel: (5826,0):dlm_wait_for_lock_mastery:1035 ERROR: status = -11
  91. Mar 4 11:24:17 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  92. Mar 4 11:24:17 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 6 from group B2F5C3291557493B99AE7326AF8B7471
  93. Mar 4 11:24:17 ams-farm kernel: (4185,3):dlm_begin_reco_handler:2669 B2F5C3291557493B99AE7326AF8B7471: dead_node previously set to 6, node 11 changing it to 6
  94. Mar 4 11:24:17 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 7 from group B2F5C3291557493B99AE7326AF8B7471
  95. Mar 4 11:24:17 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 8 from group B2F5C3291557493B99AE7326AF8B7471
  96. Mar 4 11:26:05 ams-farm kernel: o2net: connection to node node10 (num 3) at 192.168.1.10:7777 has been idle for 30.0 seconds, shutting it down.
  97. Mar 4 11:26:05 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698335.520517 now 1267698365.521543 dr 1267698335.520502 adv 1267698335.520524:1267698335.520525 func (a6c57cb2:507) 1267698265.560802:1267698265.560806)
  98. Mar 4 11:26:05 ams-farm kernel: o2net: no longer connected to node node10 (num 3) at 192.168.1.10:7777
  99. Mar 4 11:26:06 ams-farm kernel: o2net: connection to node node11 (num 0) at 192.168.1.11:7777 has been idle for 30.0 seconds, shutting it down.
  100. Mar 4 11:26:06 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698336.528279 now 1267698366.529593 dr 1267698336.528268 adv 1267698336.528280:1267698336.528281 func (a6c57cb2:507) 1267698268.575442:1267698268.575446)
  101. Mar 4 11:26:06 ams-farm kernel: o2net: no longer connected to node node11 (num 0) at 192.168.1.11:7777
  102. Mar 4 11:26:35 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors.
  103. Mar 4 11:26:36 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 0 after 30.0 seconds, giving up and returning errors.
  104. Mar 4 11:26:38 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group B2F5C3291557493B99AE7326AF8B7471
  105. Mar 4 11:26:40 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 0 from group B2F5C3291557493B99AE7326AF8B7471
  106. Mar 4 11:26:42 ams-farm kernel: (5826,0):dlm_get_lock_resource:839 B2F5C3291557493B99AE7326AF8B7471:$RECOVERY: at least one node (0) to recover before lock mastery can begin
  107. Mar 4 11:26:42 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  108. Mar 4 11:26:55 ams-farm kernel: o2net: connection to node node07 (num 11) at 192.168.1.7:7777 has been idle for 30.0 seconds, shutting it down.
  109. Mar 4 11:26:55 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267698385.389425 now 1267698415.387992 dr 1267698385.389406 adv 1267698385.389440:1267698385.389441 func (a6c57cb2:507) 1267698269.456859:1267698269.456864)
  110. Mar 4 11:26:55 ams-farm kernel: o2net: no longer connected to node node07 (num 11) at 192.168.1.7:7777
  111. Mar 4 11:26:55 ams-farm kernel: (5826,0):dlm_do_master_requery:1626 ERROR: status = -112
  112. Mar 4 11:26:55 ams-farm kernel: (5825,1):dlm_send_proxy_ast_msg:456 ERROR: status = -112
  113. Mar 4 11:26:55 ams-farm kernel: (5825,1):dlm_flush_asts:584 ERROR: status = -112
  114. Mar 4 11:26:55 ams-farm kernel: (5826,0):dlm_pre_master_reco_lockres:2159 ERROR: status = -112
  115. Mar 4 11:26:55 ams-farm kernel: (5826,0):dlm_do_master_request:1335 ERROR: link to 11 went down!
  116. Mar 4 11:26:55 ams-farm kernel: (5826,0):dlm_get_lock_resource:912 ERROR: status = -107
  117. Mar 4 11:27:25 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 11 after 30.0 seconds, giving up and returning errors.
  118. Mar 4 11:27:28 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 11 from group B2F5C3291557493B99AE7326AF8B7471
  119. Mar 4 11:27:30 ams-farm kernel: (5826,0):dlm_restart_lock_mastery:1218 ERROR: node down! 11
  120. Mar 4 11:27:30 ams-farm kernel: (5826,0):dlm_wait_for_lock_mastery:1035 ERROR: status = -11
  121. Mar 4 11:27:30 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  122. Mar 4 11:27:30 ams-farm kernel: (5826,0):dlm_do_recovery:524 (5826) Node 1 is the Recovery Master for the Dead Node 0 for Domain B2F5C3291557493B99AE7326AF8B7471
  123. Mar 4 11:27:35 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group B2F5C3291557493B99AE7326AF8B7471
  124. Mar 4 11:27:35 ams-farm kernel: (4185,3):dlm_begin_reco_handler:2669 B2F5C3291557493B99AE7326AF8B7471: dead_node previously set to 3, node 10 changing it to 3
  125. Mar 4 11:27:40 ams-farm kernel: (4185,3):o2dlm_eviction_cb:258 o2dlm has evicted node 11 from group B2F5C3291557493B99AE7326AF8B7471
  126. Mar 4 11:39:27 ams-farm kernel: o2net: connection to node node09 (num 10) at 192.168.1.9:7777 has been idle for 30.0 seconds, shutting it down.
  127. Mar 4 11:39:27 ams-farm kernel: (0,3):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1267699137.415585 now 1267699167.416915 dr 1267699137.415574 adv 1267699137.415586:1267699137.415587 func (a6c57cb2:507) 1267698469.631968:1267698469.631973)
  128. Mar 4 11:39:27 ams-farm kernel: o2net: no longer connected to node node09 (num 10) at 192.168.1.9:7777
  129. Mar 4 11:39:57 ams-farm kernel: (4185,3):o2net_connect_expired:1659 ERROR: no connection established with node 10 after 30.0 seconds, giving up and returning errors.
  130. Mar 4 11:39:58 ams-farm kernel: (5786,1):o2dlm_eviction_cb:258 o2dlm has evicted node 10 from group B2F5C3291557493B99AE7326AF8B7471
  131. Mar 4 11:39:58 ams-farm kernel: (22560,2):dlm_get_lock_resource:839 B2F5C3291557493B99AE7326AF8B7471:M000000000000000000004903d032c2: at least one node (10) to recover before lock mastery can begin
  132. Mar 4 11:39:59 ams-farm kernel: (22560,2):dlm_get_lock_resource:893 B2F5C3291557493B99AE7326AF8B7471:M000000000000000000004903d032c2: at least one node (10) to recover before lock mastery can begin
  133. Mar 4 11:40:00 ams-farm kernel: (5826,0):dlm_get_lock_resource:839 B2F5C3291557493B99AE7326AF8B7471:$RECOVERY: at least one node (10) to recover before lock mastery can begin
  134. Mar 4 11:40:00 ams-farm kernel: (5826,0):dlm_get_lock_resource:873 B2F5C3291557493B99AE7326AF8B7471: recovery map is not empty, but must master $RECOVERY lock now
  135. Mar 4 11:40:00 ams-farm kernel: (5826,0):dlm_do_recovery:524 (5826) Node 1 is the Recovery Master for the Dead Node 10 for Domain B2F5C3291557493B99AE7326AF8B7471
  136.  
  137. ##################### now it seems ocfs2 decided to clean up the mess...
  138.  
  139. Mar 4 11:40:10 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 4 from slot 1 on device (152,0)
  140. Mar 4 11:40:14 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  141. Mar 4 11:40:14 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 5 from slot 2 on device (152,0)
  142. Mar 4 11:40:18 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  143. Mar 4 11:40:18 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 6 from slot 3 on device (152,0)
  144. Mar 4 11:40:21 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  145. Mar 4 11:40:21 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 8 from slot 5 on device (152,0)
  146. Mar 4 11:40:24 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  147. Mar 4 11:40:24 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 3 from slot 9 on device (152,0)
  148. Mar 4 11:40:30 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  149. Mar 4 11:40:30 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 0 from slot 10 on device (152,0)
  150. Mar 4 11:40:33 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  151. Mar 4 11:40:33 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 11 from slot 6 on device (152,0)
  152. Mar 4 11:40:37 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  153. Mar 4 11:40:37 ams-farm kernel: (22560,1):ocfs2_replay_journal:1224 Recovering node 10 from slot 8 on device (152,0)
  154. Mar 4 11:40:40 ams-farm kernel: kjournald2 starting. Commit interval 5 seconds
  155.  
  156. ##################### here ocfs2 on the last node is not hung anymore. Now I can re-add other nodes
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!