API tools faq
paste
Guest User

Untitled

a guest
Feb 21st, 2018
57
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 83.72 KB | None | 0 0
raw download clone embed print report
  1. [root@gfs2-1 ~]# clustat
  2. Cluster Status for GFStestcluster @ Thu Oct 27 17:28:59 2011
  3. Member Status: Quorate
  4.  
  5. Member Name ID Status
  6. ------ ---- ---- ------
  7. gfs2-1 1 Online, Local
  8. gfs2-2 2 Online
  9. gfs2-3 3 Online
  10. gfs2-4 4 Online
  11. gfs2-5 5 Online
  12.  
  13.  
  14.  
  15. [root@gfs2-1 ~]# cman_tool services
  16. type level name id state
  17. fence 0 default 00010005 FAIL_ALL_STOPPED
  18. [1 2 3 4 5]
  19. dlm 1 clvmd 00020005 FAIL_ALL_STOPPED
  20. [1 2 3 4 5]
  21. dlm 1 gfsfs-rhel6 00040005 FAIL_ALL_STOPPED
  22. [1 2 3 4]
  23. gfs 2 gfsfs-rhel6 00030005 LEAVE_START_WAIT
  24. [1 2 3 4]
  25.  
  26.  
  27.  
  28.  
  29. Oct 27 17:14:16 gfs2-1 kernel: connection2:0: ping timeout of 1 secs expired, recv timeout 1, last rx 4296609091, last ping 4296610091, now 4296611091
  30. Oct 27 17:14:16 gfs2-1 kernel: connection2:0: detected conn error (1011)
  31. Oct 27 17:14:16 gfs2-1 kernel: connection1:0: ping timeout of 1 secs expired, recv timeout 1, last rx 4296609092, last ping 4296610092, now 4296611093
  32. Oct 27 17:14:16 gfs2-1 kernel: connection1:0: detected conn error (1011)
  33. Oct 27 17:14:17 gfs2-1 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  34. Oct 27 17:14:17 gfs2-1 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  35. Oct 27 17:14:17 gfs2-1 kernel: session2: session recovery timed out after 1 secs
  36. Oct 27 17:14:17 gfs2-1 kernel: session1: session recovery timed out after 1 secs
  37. Oct 27 17:14:17 gfs2-1 kernel: sd 1:0:0:1: SCSI error: return code = 0x000f0000
  38. Oct 27 17:14:17 gfs2-1 kernel: end_request: I/O error, dev sdb, sector 2118736
  39. Oct 27 17:14:17 gfs2-1 kernel: device-mapper: multipath: Failing path 8:16.
  40. Oct 27 17:14:17 gfs2-1 multipathd: dm-2: add map (uevent)
  41. Oct 27 17:14:17 gfs2-1 multipathd: dm-2: devmap already registered
  42. Oct 27 17:14:17 gfs2-1 multipathd: 8:16: mark as failed
  43. Oct 27 17:14:17 gfs2-1 multipathd: mpath3: remaining active paths: 1
  44. Oct 27 17:14:17 gfs2-1 kernel: sd 0:0:0:1: SCSI error: return code = 0x000f0000
  45. Oct 27 17:14:17 gfs2-1 kernel: end_request: I/O error, dev sda, sector 2118736
  46. Oct 27 17:14:17 gfs2-1 kernel: device-mapper: multipath: Failing path 8:0.
  47. Oct 27 17:14:17 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: gfs2_quotad: statfs error -5
  48. Oct 27 17:14:17 gfs2-1 multipathd: dm-2: add map (uevent)
  49. Oct 27 17:14:17 gfs2-1 multipathd: dm-2: devmap already registered
  50. Oct 27 17:14:18 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  51. Oct 27 17:14:18 gfs2-1 multipathd: checker failed path 8:0 in map mpath3
  52. Oct 27 17:14:18 gfs2-1 multipathd: mpath3: remaining active paths: 0
  53. Oct 27 17:14:18 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  54. Oct 27 17:14:19 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  55. Oct 27 17:14:19 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  56. Oct 27 17:14:20 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  57. Oct 27 17:14:20 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  58. Oct 27 17:14:21 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  59. Oct 27 17:14:21 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  60. Oct 27 17:14:22 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  61. Oct 27 17:14:22 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  62. Oct 27 17:14:23 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  63. Oct 27 17:14:23 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  64. Oct 27 17:14:24 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  65. Oct 27 17:14:24 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  66. Oct 27 17:14:25 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  67. Oct 27 17:14:25 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  68. Oct 27 17:14:26 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  69. Oct 27 17:14:26 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  70. Oct 27 17:14:27 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  71. Oct 27 17:14:27 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  72. Oct 27 17:14:28 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  73. Oct 27 17:14:28 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  74. Oct 27 17:14:29 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  75. Oct 27 17:14:29 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  76. Oct 27 17:14:30 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  77. Oct 27 17:14:30 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  78. Oct 27 17:14:31 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  79. Oct 27 17:14:31 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  80. Oct 27 17:14:32 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  81. Oct 27 17:14:32 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  82. Oct 27 17:14:33 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  83. Oct 27 17:14:33 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  84. Oct 27 17:14:34 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  85. Oct 27 17:14:34 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  86. Oct 27 17:14:35 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  87. Oct 27 17:14:35 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  88. Oct 27 17:14:36 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  89. Oct 27 17:14:36 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  90. Oct 27 17:14:37 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  91. Oct 27 17:14:37 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  92. Oct 27 17:14:38 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  93. Oct 27 17:14:38 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  94. Oct 27 17:14:39 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  95. Oct 27 17:14:39 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  96. Oct 27 17:14:40 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  97. Oct 27 17:14:40 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  98. Oct 27 17:14:41 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  99. Oct 27 17:14:41 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  100. Oct 27 17:14:42 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  101. Oct 27 17:14:42 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  102. Oct 27 17:14:43 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  103. Oct 27 17:14:43 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  104. Oct 27 17:14:44 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  105. Oct 27 17:14:44 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  106. Oct 27 17:14:45 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  107. Oct 27 17:14:45 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  108. Oct 27 17:14:46 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  109. Oct 27 17:14:46 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  110. Oct 27 17:14:47 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  111. Oct 27 17:14:47 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  112. Oct 27 17:14:47 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: gfs2_quotad: statfs error -5
  113. Oct 27 17:14:48 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  114. Oct 27 17:14:48 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  115. Oct 27 17:14:49 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  116. Oct 27 17:14:49 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  117. Oct 27 17:14:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  118. Oct 27 17:14:50 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  119. Oct 27 17:14:50 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  120. Oct 27 17:14:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  121. Oct 27 17:14:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  122. Oct 27 17:14:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  123. Oct 27 17:14:51 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  124. Oct 27 17:14:51 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  125. Oct 27 17:14:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  126. Oct 27 17:14:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  127. Oct 27 17:14:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  128. Oct 27 17:14:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  129. Oct 27 17:14:52 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  130. Oct 27 17:14:52 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  131. Oct 27 17:14:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  132. Oct 27 17:14:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  133. Oct 27 17:14:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  134. Oct 27 17:14:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  135. Oct 27 17:14:53 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  136. Oct 27 17:14:53 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  137. Oct 27 17:14:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  138. Oct 27 17:14:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  139. Oct 27 17:14:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  140. Oct 27 17:14:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  141. Oct 27 17:14:54 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  142. Oct 27 17:14:54 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  143. Oct 27 17:14:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  144. Oct 27 17:14:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  145. Oct 27 17:14:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  146. Oct 27 17:14:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  147. Oct 27 17:14:55 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  148. Oct 27 17:14:55 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  149. Oct 27 17:14:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  150. Oct 27 17:14:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  151. Oct 27 17:14:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  152. Oct 27 17:14:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  153. Oct 27 17:14:56 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  154. Oct 27 17:14:56 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  155. Oct 27 17:14:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  156. Oct 27 17:14:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  157. Oct 27 17:14:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  158. Oct 27 17:14:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  159. Oct 27 17:14:57 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  160. Oct 27 17:14:57 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  161. Oct 27 17:14:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  162. Oct 27 17:14:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  163. Oct 27 17:14:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  164. Oct 27 17:14:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  165. Oct 27 17:14:58 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  166. Oct 27 17:14:58 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  167. Oct 27 17:14:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  168. Oct 27 17:14:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  169. Oct 27 17:14:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  170. Oct 27 17:14:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  171. Oct 27 17:14:59 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  172. Oct 27 17:14:59 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  173. Oct 27 17:14:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  174. Oct 27 17:14:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  175. Oct 27 17:15:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  176. Oct 27 17:15:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  177. Oct 27 17:15:00 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  178. Oct 27 17:15:00 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  179. Oct 27 17:15:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  180. Oct 27 17:15:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  181. Oct 27 17:15:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  182. Oct 27 17:15:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  183. Oct 27 17:15:01 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  184. Oct 27 17:15:01 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  185. Oct 27 17:15:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  186. Oct 27 17:15:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  187. Oct 27 17:15:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  188. Oct 27 17:15:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  189. Oct 27 17:15:02 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  190. Oct 27 17:15:02 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  191. Oct 27 17:15:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  192. Oct 27 17:15:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  193. Oct 27 17:15:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  194. Oct 27 17:15:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  195. Oct 27 17:15:03 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  196. Oct 27 17:15:03 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  197. Oct 27 17:15:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  198. Oct 27 17:15:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  199. Oct 27 17:15:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  200. Oct 27 17:15:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  201. Oct 27 17:15:04 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  202. Oct 27 17:15:04 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  203. Oct 27 17:15:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  204. Oct 27 17:15:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  205. Oct 27 17:15:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  206. Oct 27 17:15:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  207. Oct 27 17:15:05 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  208. Oct 27 17:15:05 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  209. Oct 27 17:15:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  210. Oct 27 17:15:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  211. Oct 27 17:15:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  212. Oct 27 17:15:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  213. Oct 27 17:15:06 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  214. Oct 27 17:15:06 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  215. Oct 27 17:15:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  216. Oct 27 17:15:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  217. Oct 27 17:15:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  218. Oct 27 17:15:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  219. Oct 27 17:15:07 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  220. Oct 27 17:15:07 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  221. Oct 27 17:15:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  222. Oct 27 17:15:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  223. Oct 27 17:15:07 gfs2-1 iscsid: connection1:0 is operational after recovery (4 attempts)
  224. Oct 27 17:15:07 gfs2-1 iscsid: connection2:0 is operational after recovery (4 attempts)
  225. Oct 27 17:15:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  226. Oct 27 17:15:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  227. Oct 27 17:15:08 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  228. Oct 27 17:15:08 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  229. Oct 27 17:15:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  230. Oct 27 17:15:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  231. Oct 27 17:15:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  232. Oct 27 17:15:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  233. Oct 27 17:15:09 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  234. Oct 27 17:15:09 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  235. Oct 27 17:15:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  236. Oct 27 17:15:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  237. Oct 27 17:15:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  238. Oct 27 17:15:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  239. Oct 27 17:15:10 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  240. Oct 27 17:15:10 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  241. Oct 27 17:15:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  242. Oct 27 17:15:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  243. Oct 27 17:15:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  244. Oct 27 17:15:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  245. Oct 27 17:15:11 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  246. Oct 27 17:15:11 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  247. Oct 27 17:15:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  248. Oct 27 17:15:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  249. Oct 27 17:15:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  250. Oct 27 17:15:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  251. Oct 27 17:15:12 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  252. Oct 27 17:15:12 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  253. Oct 27 17:15:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  254. Oct 27 17:15:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  255. Oct 27 17:15:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  256. Oct 27 17:15:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  257. Oct 27 17:15:13 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  258. Oct 27 17:15:13 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  259. Oct 27 17:15:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  260. Oct 27 17:15:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  261. Oct 27 17:15:14 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  262. Oct 27 17:15:14 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  263. Oct 27 17:15:14 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  264. Oct 27 17:15:14 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  265. Oct 27 17:15:14 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  266. Oct 27 17:15:14 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  267. Oct 27 17:15:15 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  268. Oct 27 17:15:15 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  269. Oct 27 17:15:15 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  270. Oct 27 17:15:15 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  271. Oct 27 17:15:15 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  272. Oct 27 17:15:15 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  273. Oct 27 17:15:16 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  274. Oct 27 17:15:16 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  275. Oct 27 17:15:16 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  276. Oct 27 17:15:16 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  277. Oct 27 17:15:16 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  278. Oct 27 17:15:17 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  279. Oct 27 17:15:17 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  280. Oct 27 17:15:17 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  281. Oct 27 17:15:17 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  282. Oct 27 17:15:17 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  283. Oct 27 17:15:17 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  284. Oct 27 17:15:17 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  285. Oct 27 17:15:18 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  286. Oct 27 17:15:18 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  287. Oct 27 17:15:18 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  288. Oct 27 17:15:18 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  289. Oct 27 17:15:18 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  290. Oct 27 17:15:18 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  291. Oct 27 17:15:19 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  292. Oct 27 17:15:19 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  293. Oct 27 17:15:19 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  294. Oct 27 17:15:19 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  295. Oct 27 17:15:19 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  296. Oct 27 17:15:19 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  297. Oct 27 17:15:20 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  298. Oct 27 17:15:20 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  299. Oct 27 17:15:20 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  300. Oct 27 17:15:20 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  301. Oct 27 17:15:20 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  302. Oct 27 17:15:20 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  303. Oct 27 17:15:21 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  304. Oct 27 17:15:21 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  305. Oct 27 17:15:21 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  306. Oct 27 17:15:21 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  307. Oct 27 17:15:21 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  308. Oct 27 17:15:21 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  309. Oct 27 17:15:22 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  310. Oct 27 17:15:22 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  311. Oct 27 17:15:22 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  312. Oct 27 17:15:22 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  313. Oct 27 17:15:22 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  314. Oct 27 17:15:22 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  315. Oct 27 17:15:23 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  316. Oct 27 17:15:23 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  317. Oct 27 17:15:23 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  318. Oct 27 17:15:23 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  319. Oct 27 17:15:23 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  320. Oct 27 17:15:23 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  321. Oct 27 17:15:24 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  322. Oct 27 17:15:24 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  323. Oct 27 17:15:24 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  324. Oct 27 17:15:24 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  325. Oct 27 17:15:24 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  326. Oct 27 17:15:24 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  327. Oct 27 17:15:25 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  328. Oct 27 17:15:25 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  329. Oct 27 17:15:25 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  330. Oct 27 17:15:25 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  331. Oct 27 17:15:25 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  332. Oct 27 17:15:25 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  333. Oct 27 17:15:26 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  334. Oct 27 17:15:26 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  335. Oct 27 17:15:26 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  336. Oct 27 17:15:26 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  337. Oct 27 17:15:26 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  338. Oct 27 17:15:26 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  339. Oct 27 17:15:27 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  340. Oct 27 17:15:27 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  341. Oct 27 17:15:27 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  342. Oct 27 17:15:27 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  343. Oct 27 17:15:27 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  344. Oct 27 17:15:27 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  345. Oct 27 17:15:28 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  346. Oct 27 17:15:28 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  347. Oct 27 17:15:28 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  348. Oct 27 17:15:28 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  349. Oct 27 17:15:28 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  350. Oct 27 17:15:28 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  351. Oct 27 17:15:29 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  352. Oct 27 17:15:29 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  353. Oct 27 17:15:29 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  354. Oct 27 17:15:29 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  355. Oct 27 17:15:29 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  356. Oct 27 17:15:29 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  357. Oct 27 17:15:30 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  358. Oct 27 17:15:30 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  359. Oct 27 17:15:30 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  360. Oct 27 17:15:30 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  361. Oct 27 17:15:30 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  362. Oct 27 17:15:30 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  363. Oct 27 17:15:31 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  364. Oct 27 17:15:31 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  365. Oct 27 17:15:31 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  366. Oct 27 17:15:31 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  367. Oct 27 17:15:31 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  368. Oct 27 17:15:31 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  369. Oct 27 17:15:32 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  370. Oct 27 17:15:32 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  371. Oct 27 17:15:32 gfs2-1 kernel: connection2:0: detected conn error (1011)
  372. Oct 27 17:15:32 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  373. Oct 27 17:15:32 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  374. Oct 27 17:15:32 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  375. Oct 27 17:15:33 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  376. Oct 27 17:15:33 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  377. Oct 27 17:15:33 gfs2-1 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  378. Oct 27 17:15:33 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  379. Oct 27 17:15:33 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  380. Oct 27 17:15:33 gfs2-1 kernel: session2: session recovery timed out after 1 secs
  381. Oct 27 17:15:33 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  382. Oct 27 17:15:34 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  383. Oct 27 17:15:34 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  384. Oct 27 17:15:34 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  385. Oct 27 17:15:34 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  386. Oct 27 17:15:34 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  387. Oct 27 17:15:34 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  388. Oct 27 17:15:35 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  389. Oct 27 17:15:35 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  390. Oct 27 17:15:35 gfs2-1 iscsid: connection2:0 is operational after recovery (1 attempts)
  391. Oct 27 17:15:35 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  392. Oct 27 17:15:35 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  393. Oct 27 17:15:35 gfs2-1 kernel: connection1:0: detected conn error (1011)
  394. Oct 27 17:15:36 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  395. Oct 27 17:15:36 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  396. Oct 27 17:15:36 gfs2-1 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  397. Oct 27 17:15:36 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  398. Oct 27 17:15:36 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  399. Oct 27 17:15:37 gfs2-1 kernel: session1: session recovery timed out after 1 secs
  400. Oct 27 17:15:37 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  401. Oct 27 17:15:37 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  402. Oct 27 17:15:37 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  403. Oct 27 17:15:37 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  404. Oct 27 17:15:37 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  405. Oct 27 17:15:37 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  406. Oct 27 17:15:38 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  407. Oct 27 17:15:38 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  408. Oct 27 17:15:38 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  409. Oct 27 17:15:38 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  410. Oct 27 17:15:38 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  411. Oct 27 17:15:38 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  412. Oct 27 17:15:39 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  413. Oct 27 17:15:39 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  414. Oct 27 17:15:39 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  415. Oct 27 17:15:39 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  416. Oct 27 17:15:39 gfs2-1 iscsid: connection1:0 is operational after recovery (1 attempts)
  417. Oct 27 17:15:39 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  418. Oct 27 17:15:39 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  419. Oct 27 17:15:40 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  420. Oct 27 17:15:40 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  421. Oct 27 17:15:40 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  422. Oct 27 17:15:40 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  423. Oct 27 17:15:40 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  424. Oct 27 17:15:40 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  425. Oct 27 17:15:41 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  426. Oct 27 17:15:41 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  427. Oct 27 17:15:41 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  428. Oct 27 17:15:41 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  429. Oct 27 17:15:41 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  430. Oct 27 17:15:41 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  431. Oct 27 17:15:42 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  432. Oct 27 17:15:42 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  433. Oct 27 17:15:42 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  434. Oct 27 17:15:42 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  435. Oct 27 17:15:42 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  436. Oct 27 17:15:42 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  437. Oct 27 17:15:43 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  438. Oct 27 17:15:43 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  439. Oct 27 17:15:43 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  440. Oct 27 17:15:43 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  441. Oct 27 17:15:43 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  442. Oct 27 17:15:43 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  443. Oct 27 17:15:44 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  444. Oct 27 17:15:44 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  445. Oct 27 17:15:44 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  446. Oct 27 17:15:44 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  447. Oct 27 17:15:44 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  448. Oct 27 17:15:44 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  449. Oct 27 17:15:45 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  450. Oct 27 17:15:45 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  451. Oct 27 17:15:45 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  452. Oct 27 17:15:45 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  453. Oct 27 17:15:45 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  454. Oct 27 17:15:45 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  455. Oct 27 17:15:46 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  456. Oct 27 17:15:46 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  457. Oct 27 17:15:46 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  458. Oct 27 17:15:46 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  459. Oct 27 17:15:46 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  460. Oct 27 17:15:46 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  461. Oct 27 17:15:47 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  462. Oct 27 17:15:47 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  463. Oct 27 17:15:47 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  464. Oct 27 17:15:47 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  465. Oct 27 17:15:47 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  466. Oct 27 17:15:47 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  467. Oct 27 17:15:48 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  468. Oct 27 17:15:48 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  469. Oct 27 17:15:48 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  470. Oct 27 17:15:48 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  471. Oct 27 17:15:48 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  472. Oct 27 17:15:48 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  473. Oct 27 17:15:49 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  474. Oct 27 17:15:49 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  475. Oct 27 17:15:49 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  476. Oct 27 17:15:49 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  477. Oct 27 17:15:49 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  478. Oct 27 17:15:49 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  479. Oct 27 17:15:50 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  480. Oct 27 17:15:50 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  481. Oct 27 17:15:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  482. Oct 27 17:15:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  483. Oct 27 17:15:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  484. Oct 27 17:15:50 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  485. Oct 27 17:15:51 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  486. Oct 27 17:15:51 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  487. Oct 27 17:15:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  488. Oct 27 17:15:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  489. Oct 27 17:15:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  490. Oct 27 17:15:51 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  491. Oct 27 17:15:52 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  492. Oct 27 17:15:52 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  493. Oct 27 17:15:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  494. Oct 27 17:15:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  495. Oct 27 17:15:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  496. Oct 27 17:15:52 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  497. Oct 27 17:15:53 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  498. Oct 27 17:15:53 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  499. Oct 27 17:15:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  500. Oct 27 17:15:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  501. Oct 27 17:15:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  502. Oct 27 17:15:53 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  503. Oct 27 17:15:54 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  504. Oct 27 17:15:54 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  505. Oct 27 17:15:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  506. Oct 27 17:15:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  507. Oct 27 17:15:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  508. Oct 27 17:15:54 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  509. Oct 27 17:15:55 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  510. Oct 27 17:15:55 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  511. Oct 27 17:15:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  512. Oct 27 17:15:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  513. Oct 27 17:15:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  514. Oct 27 17:15:55 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  515. Oct 27 17:15:56 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  516. Oct 27 17:15:56 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  517. Oct 27 17:15:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  518. Oct 27 17:15:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  519. Oct 27 17:15:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  520. Oct 27 17:15:56 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  521. Oct 27 17:15:57 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  522. Oct 27 17:15:57 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  523. Oct 27 17:15:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  524. Oct 27 17:15:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  525. Oct 27 17:15:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  526. Oct 27 17:15:57 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  527. Oct 27 17:15:58 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  528. Oct 27 17:15:58 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  529. Oct 27 17:15:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  530. Oct 27 17:15:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  531. Oct 27 17:15:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  532. Oct 27 17:15:58 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  533. Oct 27 17:15:59 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  534. Oct 27 17:15:59 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  535. Oct 27 17:15:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  536. Oct 27 17:15:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  537. Oct 27 17:15:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  538. Oct 27 17:15:59 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  539. Oct 27 17:16:00 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  540. Oct 27 17:16:00 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  541. Oct 27 17:16:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  542. Oct 27 17:16:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  543. Oct 27 17:16:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  544. Oct 27 17:16:00 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  545. Oct 27 17:16:01 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  546. Oct 27 17:16:01 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  547. Oct 27 17:16:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  548. Oct 27 17:16:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  549. Oct 27 17:16:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  550. Oct 27 17:16:01 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  551. Oct 27 17:16:02 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  552. Oct 27 17:16:02 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  553. Oct 27 17:16:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  554. Oct 27 17:16:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  555. Oct 27 17:16:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  556. Oct 27 17:16:02 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  557. Oct 27 17:16:03 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  558. Oct 27 17:16:03 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  559. Oct 27 17:16:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  560. Oct 27 17:16:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  561. Oct 27 17:16:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  562. Oct 27 17:16:03 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  563. Oct 27 17:16:04 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  564. Oct 27 17:16:04 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  565. Oct 27 17:16:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  566. Oct 27 17:16:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  567. Oct 27 17:16:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  568. Oct 27 17:16:04 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  569. Oct 27 17:16:05 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  570. Oct 27 17:16:05 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  571. Oct 27 17:16:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  572. Oct 27 17:16:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  573. Oct 27 17:16:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  574. Oct 27 17:16:05 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  575. Oct 27 17:16:06 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  576. Oct 27 17:16:06 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  577. Oct 27 17:16:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  578. Oct 27 17:16:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  579. Oct 27 17:16:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  580. Oct 27 17:16:06 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  581. Oct 27 17:16:07 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  582. Oct 27 17:16:07 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  583. Oct 27 17:16:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  584. Oct 27 17:16:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  585. Oct 27 17:16:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  586. Oct 27 17:16:07 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  587. Oct 27 17:16:08 gfs2-1 kernel: connection2:0: detected conn error (1011)
  588. Oct 27 17:16:08 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  589. Oct 27 17:16:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  590. Oct 27 17:16:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  591. Oct 27 17:16:08 gfs2-1 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  592. Oct 27 17:16:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  593. Oct 27 17:16:08 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  594. Oct 27 17:16:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  595. Oct 27 17:16:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  596. Oct 27 17:16:09 gfs2-1 kernel: session2: session recovery timed out after 1 secs
  597. Oct 27 17:16:09 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  598. Oct 27 17:16:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  599. Oct 27 17:16:09 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  600. Oct 27 17:16:10 gfs2-1 kernel: connection1:0: detected conn error (1011)
  601. Oct 27 17:16:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  602. Oct 27 17:16:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  603. Oct 27 17:16:10 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  604. Oct 27 17:16:10 gfs2-1 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  605. Oct 27 17:16:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  606. Oct 27 17:16:10 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  607. Oct 27 17:16:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  608. Oct 27 17:16:11 gfs2-1 kernel: session1: session recovery timed out after 1 secs
  609. Oct 27 17:16:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  610. Oct 27 17:16:11 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  611. Oct 27 17:16:11 gfs2-1 multipathd: sdb: readsector0 checker reports path is down
  612. Oct 27 17:16:11 gfs2-1 iscsid: connection2:0 is operational after recovery (1 attempts)
  613. Oct 27 17:16:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  614. Oct 27 17:16:11 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  615. Oct 27 17:16:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  616. Oct 27 17:16:12 gfs2-1 multipathd: sda: readsector0 checker reports path is down
  617. Oct 27 17:16:12 gfs2-1 multipathd: sdb: readsector0 checker reports path is up
  618. Oct 27 17:16:12 gfs2-1 multipathd: 8:16: reinstated
  619. Oct 27 17:16:12 gfs2-1 multipathd: mpath3: remaining active paths: 1
  620. Oct 27 17:16:12 gfs2-1 multipathd: dm-2: add map (uevent)
  621. Oct 27 17:16:12 gfs2-1 multipathd: dm-2: devmap already registered
  622. Oct 27 17:16:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Failed
  623. Oct 27 17:16:12 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Trying to acquire journal lock...
  624. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Looking at journal...
  625. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Acquiring the transaction lock...
  626. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Replaying journal...
  627. Oct 27 17:16:13 gfs2-1 multipathd: sda: readsector0 checker reports path is up
  628. Oct 27 17:16:13 gfs2-1 multipathd: 8:0: reinstated
  629. Oct 27 17:16:13 gfs2-1 multipathd: mpath3: remaining active paths: 2
  630. Oct 27 17:16:13 gfs2-1 multipathd: dm-2: add map (uevent)
  631. Oct 27 17:16:13 gfs2-1 multipathd: dm-2: devmap already registered
  632. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Replayed 1 of 1 blocks
  633. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Found 0 revoke tags
  634. Oct 27 17:16:13 gfs2-1 iscsid: connection1:0 is operational after recovery (1 attempts)
  635. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Journal replayed in 1s
  636. Oct 27 17:16:13 gfs2-1 kernel: GFS2: fsid=GFStestcluster:gfsfs-rhel6.3: jid=0: Done
  637. Oct 27 17:17:53 gfs2-1 kernel: INFO: task gfs2_quotad:2160 blocked for more than 120 seconds.
  638. Oct 27 17:17:53 gfs2-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  639. Oct 27 17:17:53 gfs2-1 kernel: gfs2_quotad D ffff810001036420 0 2160 11 2159 (L-TLB)
  640. Oct 27 17:17:53 gfs2-1 kernel: ffff8100101abcd0 0000000000000046 ffffffff80311da0 0000000000000000
  641. Oct 27 17:17:53 gfs2-1 kernel: ffff8100101abc70 000000000000000a ffff81000f8bb860 ffffffff80308b60
  642. Oct 27 17:17:53 gfs2-1 kernel: 00270547a119f14f 0000000000002636 ffff81000f8bba48 0000000000000001
  643. Oct 27 17:17:53 gfs2-1 kernel: Call Trace:
  644. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  645. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff88669eec>] :gfs2:just_schedule+0x9/0xe
  646. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  647. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  648. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  649. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff800a0a02>] wake_bit_function+0x0/0x23
  650. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff88669ede>] :gfs2:gfs2_glock_wait+0x2b/0x30
  651. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff886805ee>] :gfs2:gfs2_statfs_sync+0x3f/0x165
  652. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff886805e6>] :gfs2:gfs2_statfs_sync+0x37/0x165
  653. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
  654. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8867a3df>] :gfs2:quotad_check_timeo+0x20/0x60
  655. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8867bedc>] :gfs2:gfs2_quotad+0xde/0x214
  656. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff800a09d4>] autoremove_wake_function+0x0/0x2e
  657. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8867bdfe>] :gfs2:gfs2_quotad+0x0/0x214
  658. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  659. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8003290a>] kthread+0xfe/0x132
  660. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
  661. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  662. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8003280c>] kthread+0x0/0x132
  663. Oct 27 17:17:53 gfs2-1 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
  664. Oct 27 17:17:53 gfs2-1 kernel:
  665. Oct 27 17:19:53 gfs2-1 kernel: INFO: task gfs2_quotad:2160 blocked for more than 120 seconds.
  666. Oct 27 17:19:53 gfs2-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  667. Oct 27 17:19:53 gfs2-1 kernel: gfs2_quotad D ffff810001036420 0 2160 11 2159 (L-TLB)
  668. Oct 27 17:19:53 gfs2-1 kernel: ffff8100101abcd0 0000000000000046 ffffffff80311da0 0000000000000000
  669. Oct 27 17:19:53 gfs2-1 kernel: ffff8100101abc70 000000000000000a ffff81000f8bb860 ffffffff80308b60
  670. Oct 27 17:19:53 gfs2-1 kernel: 00270547a119f14f 0000000000002636 ffff81000f8bba48 0000000000000001
  671. Oct 27 17:19:53 gfs2-1 kernel: Call Trace:
  672. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  673. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff88669eec>] :gfs2:just_schedule+0x9/0xe
  674. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  675. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  676. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  677. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff800a0a02>] wake_bit_function+0x0/0x23
  678. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff88669ede>] :gfs2:gfs2_glock_wait+0x2b/0x30
  679. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff886805ee>] :gfs2:gfs2_statfs_sync+0x3f/0x165
  680. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff886805e6>] :gfs2:gfs2_statfs_sync+0x37/0x165
  681. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
  682. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8867a3df>] :gfs2:quotad_check_timeo+0x20/0x60
  683. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8867bedc>] :gfs2:gfs2_quotad+0xde/0x214
  684. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff800a09d4>] autoremove_wake_function+0x0/0x2e
  685. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8867bdfe>] :gfs2:gfs2_quotad+0x0/0x214
  686. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  687. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8003290a>] kthread+0xfe/0x132
  688. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
  689. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  690. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8003280c>] kthread+0x0/0x132
  691. Oct 27 17:19:53 gfs2-1 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
  692. Oct 27 17:19:53 gfs2-1 kernel:
  693. Oct 27 17:21:41 gfs2-1 openais[1979]: [TOTEM] The token was lost in the OPERATIONAL state.
  694. Oct 27 17:21:41 gfs2-1 openais[1979]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
  695. Oct 27 17:21:41 gfs2-1 openais[1979]: [TOTEM] Transmit multicast socket send buffer size (258048 bytes).
  696. Oct 27 17:21:41 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 2.
  697. Oct 27 17:21:53 gfs2-1 kernel: INFO: task gfs2_quotad:2160 blocked for more than 120 seconds.
  698. Oct 27 17:21:53 gfs2-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  699. Oct 27 17:21:53 gfs2-1 kernel: gfs2_quotad D ffff810001036420 0 2160 11 2159 (L-TLB)
  700. Oct 27 17:21:53 gfs2-1 kernel: ffff8100101abcd0 0000000000000046 ffffffff80311da0 0000000000000000
  701. Oct 27 17:21:53 gfs2-1 kernel: ffff8100101abc70 000000000000000a ffff81000f8bb860 ffffffff80308b60
  702. Oct 27 17:21:53 gfs2-1 kernel: 00270547a119f14f 0000000000002636 ffff81000f8bba48 0000000000000001
  703. Oct 27 17:21:53 gfs2-1 kernel: Call Trace:
  704. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  705. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff88669eec>] :gfs2:just_schedule+0x9/0xe
  706. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  707. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  708. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  709. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff800a0a02>] wake_bit_function+0x0/0x23
  710. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff88669ede>] :gfs2:gfs2_glock_wait+0x2b/0x30
  711. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff886805ee>] :gfs2:gfs2_statfs_sync+0x3f/0x165
  712. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff886805e6>] :gfs2:gfs2_statfs_sync+0x37/0x165
  713. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
  714. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8867a3df>] :gfs2:quotad_check_timeo+0x20/0x60
  715. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8867bedc>] :gfs2:gfs2_quotad+0xde/0x214
  716. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff800a09d4>] autoremove_wake_function+0x0/0x2e
  717. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8867bdfe>] :gfs2:gfs2_quotad+0x0/0x214
  718. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  719. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8003290a>] kthread+0xfe/0x132
  720. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
  721. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  722. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8003280c>] kthread+0x0/0x132
  723. Oct 27 17:21:53 gfs2-1 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
  724. Oct 27 17:21:53 gfs2-1 kernel:
  725. Oct 27 17:21:53 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 11.
  726. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] Creating commit token because I am the rep.
  727. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] Saving state aru a6 high seq received a6
  728. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] Storing new sequence id for ring 748
  729. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] entering COMMIT state.
  730. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] entering RECOVERY state.
  731. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] position [0] member 192.168.122.30:
  732. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1860 rep 192.168.122.30
  733. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] aru a6 high delivered a6 received flag 1
  734. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] position [1] member 192.168.122.31:
  735. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1860 rep 192.168.122.30
  736. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] aru a6 high delivered a6 received flag 1
  737. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] position [2] member 192.168.122.32:
  738. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1860 rep 192.168.122.30
  739. Oct 27 17:21:54 gfs2-1 kernel: dlm: closing connection to node 5
  740. Oct 27 17:21:54 gfs2-1 fenced[1998]: gfs2-5 not a cluster member after 0 sec post_fail_delay
  741. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] aru a6 high delivered a6 received flag 1
  742. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] position [3] member 192.168.122.33:
  743. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1860 rep 192.168.122.30
  744. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] aru a6 high delivered a6 received flag 1
  745. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] Did not need to originate any messages in recovery.
  746. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] Sending initial ORF token
  747. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  748. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] New Configuration:
  749. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  750. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  751. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  752. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  753. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] Members Left:
  754. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  755. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] Members Joined:
  756. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  757. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] New Configuration:
  758. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  759. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  760. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  761. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  762. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] Members Left:
  763. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] Members Joined:
  764. Oct 27 17:21:54 gfs2-1 openais[1979]: [SYNC ] This node is within the primary component and will provide service.
  765. Oct 27 17:21:54 gfs2-1 openais[1979]: [TOTEM] entering OPERATIONAL state.
  766. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.30
  767. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.31
  768. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.32
  769. Oct 27 17:21:54 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.33
  770. Oct 27 17:21:54 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 4
  771. Oct 27 17:21:55 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 1
  772. Oct 27 17:21:55 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 2
  773. Oct 27 17:21:55 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 3
  774. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 11.
  775. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] Creating commit token because I am the rep.
  776. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] Saving state aru 44 high seq received 44
  777. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] Storing new sequence id for ring 74c
  778. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] entering COMMIT state.
  779. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] entering RECOVERY state.
  780. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] position [0] member 192.168.122.30:
  781. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1864 rep 192.168.122.30
  782. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] aru 44 high delivered 44 received flag 1
  783. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] position [1] member 192.168.122.31:
  784. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1864 rep 192.168.122.30
  785. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] aru 44 high delivered 44 received flag 1
  786. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] position [2] member 192.168.122.32:
  787. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1864 rep 192.168.122.30
  788. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] aru 44 high delivered 44 received flag 1
  789. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] position [3] member 192.168.122.33:
  790. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1864 rep 192.168.122.30
  791. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] aru 44 high delivered 44 received flag 1
  792. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] position [4] member 192.168.122.34:
  793. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1864 rep 192.168.122.34
  794. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] aru c high delivered c received flag 1
  795. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] Did not need to originate any messages in recovery.
  796. Oct 27 17:22:23 gfs2-1 openais[1979]: [TOTEM] Sending initial ORF token
  797. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  798. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] New Configuration:
  799. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  800. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  801. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  802. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  803. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] Members Left:
  804. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] Members Joined:
  805. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  806. Oct 27 17:22:24 gfs2-1 openais[1979]: [CLM ] New Configuration:
  807. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  808. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  809. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  810. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  811. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  812. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] Members Left:
  813. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] Members Joined:
  814. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  815. Oct 27 17:22:25 gfs2-1 openais[1979]: [SYNC ] This node is within the primary component and will provide service.
  816. Oct 27 17:22:25 gfs2-1 openais[1979]: [TOTEM] entering OPERATIONAL state.
  817. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.30
  818. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.31
  819. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.32
  820. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.33
  821. Oct 27 17:22:25 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.34
  822. Oct 27 17:22:25 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 1
  823. Oct 27 17:22:25 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 2
  824. Oct 27 17:22:25 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 3
  825. Oct 27 17:22:25 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 4
  826. Oct 27 17:22:31 gfs2-1 kernel: dlm: got connection from 5
  827. Oct 27 17:23:41 gfs2-1 avahi-daemon[2441]: Invalid query packet.
  828. Oct 27 17:23:48 gfs2-1 last message repeated 16 times
  829. Oct 27 17:23:53 gfs2-1 kernel: INFO: task gfs2_quotad:2160 blocked for more than 120 seconds.
  830. Oct 27 17:23:53 gfs2-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  831. Oct 27 17:23:53 gfs2-1 kernel: gfs2_quotad D ffff810001036420 0 2160 11 2159 (L-TLB)
  832. Oct 27 17:23:53 gfs2-1 kernel: ffff8100101abcd0 0000000000000046 ffffffff80311da0 0000000000000000
  833. Oct 27 17:23:53 gfs2-1 kernel: ffff8100101abc70 000000000000000a ffff81000f8bb860 ffffffff80308b60
  834. Oct 27 17:23:53 gfs2-1 kernel: 00270547a119f14f 0000000000002636 ffff81000f8bba48 0000000000000001
  835. Oct 27 17:23:53 gfs2-1 kernel: Call Trace:
  836. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  837. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff88669eec>] :gfs2:just_schedule+0x9/0xe
  838. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  839. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  840. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  841. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff800a0a02>] wake_bit_function+0x0/0x23
  842. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff88669ede>] :gfs2:gfs2_glock_wait+0x2b/0x30
  843. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff886805ee>] :gfs2:gfs2_statfs_sync+0x3f/0x165
  844. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff886805e6>] :gfs2:gfs2_statfs_sync+0x37/0x165
  845. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
  846. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8867a3df>] :gfs2:quotad_check_timeo+0x20/0x60
  847. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8867bedc>] :gfs2:gfs2_quotad+0xde/0x214
  848. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff800a09d4>] autoremove_wake_function+0x0/0x2e
  849. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8867bdfe>] :gfs2:gfs2_quotad+0x0/0x214
  850. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  851. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8003290a>] kthread+0xfe/0x132
  852. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
  853. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  854. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8003280c>] kthread+0x0/0x132
  855. Oct 27 17:23:53 gfs2-1 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
  856. Oct 27 17:23:53 gfs2-1 kernel:
  857. Oct 27 17:23:58 gfs2-1 avahi-daemon[2441]: Invalid query packet.
  858. Oct 27 17:23:58 gfs2-1 avahi-daemon[2441]: Invalid query packet.
  859. Oct 27 17:24:15 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 12.
  860. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 11.
  861. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] Creating commit token because I am the rep.
  862. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] Saving state aru 7b high seq received 7b
  863. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] Storing new sequence id for ring 750
  864. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] entering COMMIT state.
  865. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] entering RECOVERY state.
  866. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] position [0] member 192.168.122.30:
  867. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1868 rep 192.168.122.30
  868. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] aru 7b high delivered 7b received flag 1
  869. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] position [1] member 192.168.122.31:
  870. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1868 rep 192.168.122.30
  871. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] aru 7b high delivered 7b received flag 1
  872. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] position [2] member 192.168.122.32:
  873. Oct 27 17:24:27 gfs2-1 kernel: dlm: closing connection to node 4
  874. Oct 27 17:24:27 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1868 rep 192.168.122.30
  875. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] aru 7b high delivered 7b received flag 1
  876. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] position [3] member 192.168.122.34:
  877. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1868 rep 192.168.122.30
  878. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] aru 7b high delivered 7b received flag 1
  879. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] Did not need to originate any messages in recovery.
  880. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] Sending initial ORF token
  881. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  882. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] New Configuration:
  883. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  884. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  885. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  886. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  887. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] Members Left:
  888. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  889. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] Members Joined:
  890. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  891. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] New Configuration:
  892. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  893. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  894. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  895. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  896. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] Members Left:
  897. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] Members Joined:
  898. Oct 27 17:24:28 gfs2-1 openais[1979]: [SYNC ] This node is within the primary component and will provide service.
  899. Oct 27 17:24:28 gfs2-1 openais[1979]: [TOTEM] entering OPERATIONAL state.
  900. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.30
  901. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.31
  902. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.32
  903. Oct 27 17:24:28 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.34
  904. Oct 27 17:24:28 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 5
  905. Oct 27 17:24:28 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 1
  906. Oct 27 17:24:28 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 2
  907. Oct 27 17:24:28 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 3
  908. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] entering GATHER state from 11.
  909. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] Creating commit token because I am the rep.
  910. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] Saving state aru 3e high seq received 3e
  911. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] Storing new sequence id for ring 754
  912. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] entering COMMIT state.
  913. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] entering RECOVERY state.
  914. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] position [0] member 192.168.122.30:
  915. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1872 rep 192.168.122.30
  916. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] aru 3e high delivered 3e received flag 1
  917. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] position [1] member 192.168.122.31:
  918. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1872 rep 192.168.122.30
  919. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] aru 3e high delivered 3e received flag 1
  920. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] position [2] member 192.168.122.32:
  921. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1872 rep 192.168.122.30
  922. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] aru 3e high delivered 3e received flag 1
  923. Oct 27 17:24:58 gfs2-1 openais[1979]: [TOTEM] position [3] member 192.168.122.33:
  924. Oct 27 17:24:59 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1872 rep 192.168.122.33
  925. Oct 27 17:24:59 gfs2-1 openais[1979]: [TOTEM] aru c high delivered c received flag 1
  926. Oct 27 17:25:00 gfs2-1 openais[1979]: [TOTEM] position [4] member 192.168.122.34:
  927. Oct 27 17:25:00 gfs2-1 openais[1979]: [TOTEM] previous ring seq 1872 rep 192.168.122.30
  928. Oct 27 17:25:00 gfs2-1 openais[1979]: [TOTEM] aru 3e high delivered 3e received flag 1
  929. Oct 27 17:25:00 gfs2-1 openais[1979]: [TOTEM] Did not need to originate any messages in recovery.
  930. Oct 27 17:25:00 gfs2-1 openais[1979]: [TOTEM] Sending initial ORF token
  931. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  932. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] New Configuration:
  933. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  934. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  935. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  936. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  937. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] Members Left:
  938. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] Members Joined:
  939. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] CLM CONFIGURATION CHANGE
  940. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] New Configuration:
  941. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.30)
  942. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.31)
  943. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.32)
  944. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  945. Oct 27 17:25:01 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.34)
  946. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] Members Left:
  947. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] Members Joined:
  948. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] r(0) ip(192.168.122.33)
  949. Oct 27 17:25:02 gfs2-1 openais[1979]: [SYNC ] This node is within the primary component and will provide service.
  950. Oct 27 17:25:02 gfs2-1 openais[1979]: [TOTEM] entering OPERATIONAL state.
  951. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.30
  952. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.31
  953. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.32
  954. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.33
  955. Oct 27 17:25:02 gfs2-1 openais[1979]: [CLM ] got nodejoin message 192.168.122.34
  956. Oct 27 17:25:02 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 5
  957. Oct 27 17:25:02 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 1
  958. Oct 27 17:25:02 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 2
  959. Oct 27 17:25:02 gfs2-1 openais[1979]: [CPG ] got joinlist message from node 3
  960. Oct 27 17:25:53 gfs2-1 kernel: INFO: task gfs2_quotad:2160 blocked for more than 120 seconds.
  961. Oct 27 17:25:53 gfs2-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  962. Oct 27 17:25:53 gfs2-1 kernel: gfs2_quotad D ffff810001036420 0 2160 11 2159 (L-TLB)
  963. Oct 27 17:25:53 gfs2-1 kernel: ffff8100101abcd0 0000000000000046 ffffffff80311da0 0000000000000000
  964. Oct 27 17:25:53 gfs2-1 kernel: ffff8100101abc70 000000000000000a ffff81000f8bb860 ffffffff80308b60
  965. Oct 27 17:25:53 gfs2-1 kernel: 00270547a119f14f 0000000000002636 ffff81000f8bba48 0000000000000001
  966. Oct 27 17:25:53 gfs2-1 kernel: Call Trace:
  967. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  968. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff88669eec>] :gfs2:just_schedule+0x9/0xe
  969. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  970. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff88669ee3>] :gfs2:just_schedule+0x0/0xe
  971. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  972. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff800a0a02>] wake_bit_function+0x0/0x23
  973. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff88669ede>] :gfs2:gfs2_glock_wait+0x2b/0x30
  974. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff886805ee>] :gfs2:gfs2_statfs_sync+0x3f/0x165
  975. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff886805e6>] :gfs2:gfs2_statfs_sync+0x37/0x165
  976. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
  977. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8867a3df>] :gfs2:quotad_check_timeo+0x20/0x60
  978. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8867bedc>] :gfs2:gfs2_quotad+0xde/0x214
  979. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff800a09d4>] autoremove_wake_function+0x0/0x2e
  980. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8867bdfe>] :gfs2:gfs2_quotad+0x0/0x214
  981. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  982. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8003290a>] kthread+0xfe/0x132
  983. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
  984. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff800a07bc>] keventd_create_kthread+0x0/0xc4
  985. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8003280c>] kthread+0x0/0x132
  986. Oct 27 17:25:53 gfs2-1 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
  987. Oct 27 17:25:53 gfs2-1 kernel:
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!