Advertisement
Guest User

hv-011-02

a guest
Jun 12th, 2012
152
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 17.34 KB | None | 0 0
  1. May 10 05:09:30 hv011 kernel: connection3:0: ping timeout of 10 secs expired, recv timeout 5, last rx 6123605368, last ping 6123606618, now 6123611587
  2. May 10 05:09:30 hv011 kernel: connection3:0: detected conn error (1011)
  3. May 10 05:09:36 hv011 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  4. May 10 05:09:36 hv011 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  5. May 10 05:09:41 hv011 iscsid: Kernel reported iSCSI connection 4:0 error (1011) state (3)
  6. May 10 05:09:50 hv011 kernel: session1: session recovery timed out after 15 secs
  7. May 10 05:09:50 hv011 kernel: sd 4:0:0:1: SCSI error: return code = 0x000f0000
  8. May 10 05:09:50 hv011 kernel: end_request: I/O error, dev sdf, sector 2019871776
  9. May 10 05:10:16 hv011 nrpe[5855]: Error: Could not complete SSL handshake. 5
  10. May 10 05:10:20 hv011 kernel: device-mapper: multipath: Failing path 8:80.
  11. May 10 05:10:22 hv011 kernel: session2: session recovery timed out after 15 secs
  12. May 10 05:10:27 hv011 multipathd: sdt: readsector0 checker reports path is down
  13. May 10 05:10:28 hv011 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  14. May 10 05:10:39 hv011 multipathd: checker failed path 65:48 in map mpath9
  15. May 10 05:10:39 hv011 multipathd: mpath9: remaining active paths: 0
  16. May 10 05:10:39 hv011 kernel: end_request: I/O error, dev sdh, sector 2019871776
  17. May 10 05:10:39 hv011 multipathd: dm-4: add map (uevent)
  18. May 10 05:10:39 hv011 kernel: device-mapper: multipath: Failing path 8:112.
  19. May 10 05:10:39 hv011 multipathd: dm-4: devmap already registered
  20. May 10 05:10:39 hv011 kernel: BUG: soft lockup - CPU#7 stuck for 60s! [ksoftirqd/7:24]
  21. May 10 05:10:39 hv011 multipathd: dm-4: add map (uevent)
  22. May 10 05:10:39 hv011 kernel: CPU 7:
  23. May 10 05:10:39 hv011 multipathd: dm-4: devmap already registered
  24. May 10 05:10:39 hv011 kernel: Modules linked in: xt_multiport ext4 jbd2 crc16 nls_utf8 hfsplus fuse tun sch_tbf xt_state ip_conntrack nfnetlink xt_physdev nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sr_mod cdrom sg pcspkr shpchp i2c_i801 i2c_core igb 8021q dca serial_core tpm_tis tpm tpm_bios serio_raw dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix li
  25. May 10 05:10:39 hv011 multipathd: 8:80: mark as failed
  26. May 10 05:10:39 hv011 kernel: ata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  27. May 10 05:10:39 hv011 multipathd: mpath3: remaining active paths: 0
  28. May 10 05:10:39 hv011 kernel: Pid: 24, comm: ksoftirqd/7 Not tainted 2.6.18-274.18.1.el5xen #1
  29. May 10 05:10:39 hv011 multipathd: 8:112: mark as failed
  30. May 10 05:10:39 hv011 kernel: RIP: e030:[<ffffffff802610c1>] [<ffffffff802610c1>] __write_lock_failed+0x9/0x20
  31. May 10 05:10:39 hv011 multipathd: dm-10: add map (uevent)
  32. May 10 05:10:39 hv011 kernel: RSP: e02b:ffff880007ba3f10 EFLAGS: 00000287
  33. May 10 05:10:39 hv011 multipathd: dm-10: devmap already registered
  34. May 10 05:10:39 hv011 kernel: RAX: ffff880007b8ffd8 RBX: ffffffff8882f660 RCX: 0000000000000000
  35. May 10 05:10:39 hv011 multipathd: 8:112: mark as failed
  36. May 10 05:10:39 hv011 kernel: RDX: ffffffffff5781c0 RSI: ffff880079691dc8 RDI: ffffffff8882f660
  37. May 10 05:10:39 hv011 multipathd: sdf: readsector0 checker reports path is down
  38. May 10 05:10:39 hv011 kernel: RBP: ffff880007b90000 R08: ffff880079691c18 R09: 0000000000000007
  39. May 10 05:10:39 hv011 multipathd: sdf: readsector0 checker reports path is down
  40. May 10 05:10:39 hv011 kernel: R10: ffff88001c7a59e8 R11: ffff88001c7a5a00 R12: ffff88002db2dec0
  41. May 10 05:10:39 hv011 multipathd: sdb: readsector0 checker reports path is down
  42. May 10 05:10:39 hv011 kernel: R13: 0000000000000100 R14: ffff88002db2dea8 R15: ffffffff888230fd
  43. May 10 05:10:39 hv011 multipathd: checker failed path 8:16 in map mpath1
  44. May 10 05:10:39 hv011 kernel: FS: 00002b7bc6789b10(0000) GS:ffffffff80631380(0000) knlGS:0000000000000000
  45. May 10 05:10:29 hv011 iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3)
  46. May 10 05:10:39 hv011 multipathd: mpath1: remaining active paths: 0
  47. May 10 05:10:39 hv011 kernel: CS: e033 DS: 0000 ES: 0000
  48. May 10 05:10:39 hv011 multipathd: dm-2: add map (uevent)
  49. May 10 05:10:39 hv011 kernel:
  50. May 10 05:10:39 hv011 multipathd: dm-2: devmap already registered
  51. May 10 05:10:39 hv011 kernel: Call Trace:
  52. May 10 05:10:39 hv011 kernel: <IRQ> [<ffffffff80263a31>] _write_lock_bh+0x1a/0x1c
  53. May 10 05:10:39 hv011 kernel: [<ffffffff8882310d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  54. May 10 05:10:39 hv011 kernel: [<ffffffff80294199>] run_timer_softirq+0x191/0x242
  55. May 10 05:10:39 hv011 kernel: [<ffffffff80212f08>] __do_softirq+0x8d/0x13b
  56. May 10 05:10:39 hv011 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  57. May 10 05:10:39 hv011 kernel: <EOI> [<ffffffff80290ea7>] ksoftirqd+0x0/0xbf
  58. May 10 05:10:39 hv011 kernel: [<ffffffff8026db89>] do_softirq+0x31/0x90
  59. May 10 05:10:39 hv011 kernel: [<ffffffff80290f06>] ksoftirqd+0x5f/0xbf
  60. May 10 05:10:39 hv011 kernel: [<ffffffff802339b7>] kthread+0xfe/0x132
  61. May 10 05:10:39 hv011 kernel: [<ffffffff8025fb2c>] child_rip+0xa/0x12
  62. May 10 05:10:39 hv011 kernel: [<ffffffff802338b9>] kthread+0x0/0x132
  63. May 10 05:10:39 hv011 kernel: [<ffffffff8025fb22>] child_rip+0x0/0x12
  64. May 10 05:10:39 hv011 kernel:
  65. May 10 05:10:39 hv011 kernel: sd 7:0:0:6: timing out command, waited 60s
  66. May 10 05:10:39 hv011 kernel: device-mapper: multipath: Failing path 65:48.
  67. May 10 05:10:39 hv011 kernel: device-mapper: multipath: Failing path 8:16.
  68. May 10 05:10:41 hv011 multipathd: sdf: readsector0 checker reports path is down
  69. May 10 05:10:43 hv011 kernel: session4: session recovery timed out after 15 secs
  70. May 10 05:10:43 hv011 kernel: session3: session recovery timed out after 15 secs
  71. May 10 05:10:43 hv011 kernel: sd 7:0:0:1: SCSI error: return code = 0x000f0000
  72. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdg, sector 3220351344
  73. May 10 05:10:43 hv011 kernel: device-mapper: multipath: Failing path 8:96.
  74. May 10 05:10:43 hv011 kernel: sd 7:0:0:1: SCSI error: return code = 0x000f0000
  75. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdg, sector 3897469752
  76. May 10 05:10:43 hv011 kernel: sd 7:0:0:2: SCSI error: return code = 0x000f0000
  77. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdj, sector 556520592
  78. May 10 05:10:43 hv011 kernel: device-mapper: multipath: Failing path 8:144.
  79. May 10 05:10:43 hv011 kernel: sd 7:0:0:3: SCSI error: return code = 0x000f0000
  80. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdl, sector 69222184
  81. May 10 05:10:43 hv011 kernel: device-mapper: multipath: Failing path 8:176.
  82. May 10 05:10:43 hv011 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  83. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdo, sector 1664321472
  84. May 10 05:10:43 hv011 kernel: device-mapper: multipath: Failing path 8:224.
  85. May 10 05:10:43 hv011 multipathd: dm-5: add map (uevent)
  86. May 10 05:10:43 hv011 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  87. May 10 05:10:43 hv011 multipathd: dm-5: devmap already registered
  88. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdo, sector 1586542736
  89. May 10 05:10:43 hv011 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  90. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdo, sector 1664319872
  91. May 10 05:10:43 hv011 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  92. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdo, sector 1359054496
  93. May 10 05:10:43 hv011 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  94. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdo, sector 1664320832
  95. May 10 05:10:43 hv011 kernel: sd 7:0:0:6: SCSI error: return code = 0x000f0000
  96. May 10 05:10:43 hv011 kernel: end_request: I/O error, dev sdt, sector 211467128
  97. May 10 05:10:44 hv011 multipathd: 8:96: mark as failed
  98. May 10 05:10:44 hv011 multipathd: mpath4: remaining active paths: 0
  99. May 10 05:10:44 hv011 multipathd: 8:128: mark as failed
  100. May 10 05:10:44 hv011 multipathd: 8:176: mark as failed
  101. May 10 05:10:44 hv011 multipathd: mpath6: remaining active paths: 0
  102. May 10 05:10:44 hv011 multipathd: 8:192: mark as failed
  103. May 10 05:10:44 hv011 multipathd: dm-6: add map (uevent)
  104. May 10 05:10:44 hv011 multipathd: dm-6: devmap already registered
  105. May 10 05:10:44 hv011 multipathd: dm-7: add map (uevent)
  106. May 10 05:10:44 hv011 multipathd: dm-7: devmap already registered
  107. May 10 05:10:44 hv011 multipathd: dm-8: add map (uevent)
  108. May 10 05:10:44 hv011 multipathd: dm-8: devmap already registered
  109. May 10 05:10:44 hv011 multipathd: 8:144: mark as failed
  110. May 10 05:10:44 hv011 kernel: sd 6:0:0:2: SCSI error: return code = 0x000f0000
  111. May 10 05:10:44 hv011 multipathd: mpath5: remaining active paths: 0
  112. May 10 05:10:44 hv011 kernel: end_request: I/O error, dev sdk, sector 556520592
  113. May 10 05:10:44 hv011 kernel: device-mapper: multipath: Failing path 8:160.
  114. May 10 05:10:44 hv011 kernel: sd 6:0:0:6: SCSI error: return code = 0x000f0000
  115. May 10 05:10:44 hv011 kernel: end_request: I/O error, dev sdq, sector 211467128
  116. May 10 05:10:44 hv011 kernel: device-mapper: multipath: Failing path 65:0.
  117. May 10 05:10:44 hv011 multipathd: 65:0: mark as failed
  118. May 10 05:10:44 hv011 multipathd: dm-6: add map (uevent)
  119. May 10 05:10:44 hv011 multipathd: dm-6: devmap already registered
  120. May 10 05:10:44 hv011 multipathd: dm-10: add map (uevent)
  121. May 10 05:10:44 hv011 multipathd: dm-10: devmap already registered
  122. May 10 05:10:44 hv011 multipathd: 8:208: mark as failed
  123. May 10 05:10:44 hv011 multipathd: 8:224: mark as failed
  124. May 10 05:10:44 hv011 multipathd: mpath7: remaining active paths: 0
  125. May 10 05:10:44 hv011 multipathd: sde: readsector0 checker reports path is down
  126. May 10 05:10:44 hv011 multipathd: checker failed path 8:64 in map mpath2
  127. May 10 05:10:44 hv011 kernel: device-mapper: multipath: Failing path 8:64.
  128. May 10 05:10:44 hv011 multipathd: mpath2: remaining active paths: 0
  129. May 10 05:10:44 hv011 multipathd: sdg: readsector0 checker reports path is down
  130. May 10 05:10:44 hv011 multipathd: sdj: readsector0 checker reports path is down
  131. May 10 05:10:44 hv011 multipathd: sdr: readsector0 checker reports path is down
  132. May 10 05:10:44 hv011 multipathd: checker failed path 65:16 in map mpath8
  133. May 10 05:10:44 hv011 kernel: device-mapper: multipath: Failing path 65:16.
  134. May 10 05:10:44 hv011 multipathd: mpath8: remaining active paths: 0
  135. May 10 05:10:44 hv011 multipathd: sdt: readsector0 checker reports path is down
  136. May 10 05:10:44 hv011 multipathd: sdu: readsector0 checker reports path is down
  137. May 10 05:10:44 hv011 multipathd: checker failed path 65:64 in map mpath10
  138. May 10 05:10:44 hv011 kernel: device-mapper: multipath: Failing path 65:64.
  139. May 10 05:10:45 hv011 multipathd: mpath10: remaining active paths: 0
  140. May 10 05:10:45 hv011 multipathd: sdb: readsector0 checker reports path is down
  141. May 10 05:10:45 hv011 multipathd: dm-3: add map (uevent)
  142. May 10 05:10:45 hv011 multipathd: dm-3: devmap already registered
  143. May 10 05:10:45 hv011 multipathd: dm-9: add map (uevent)
  144. May 10 05:10:45 hv011 multipathd: dm-9: devmap already registered
  145. May 10 05:10:45 hv011 multipathd: dm-11: add map (uevent)
  146. May 10 05:10:45 hv011 multipathd: dm-11: devmap already registered
  147. May 10 05:10:45 hv011 multipathd: 8:160: mark as failed
  148. May 10 05:10:46 hv011 multipathd: sdf: readsector0 checker reports path is down
  149. May 10 05:10:48 hv011 multipathd: sdl: readsector0 checker reports path is down
  150. May 10 05:10:48 hv011 multipathd: sdo: readsector0 checker reports path is down
  151. May 10 05:10:49 hv011 multipathd: sde: readsector0 checker reports path is down
  152. May 10 05:10:49 hv011 multipathd: sdg: readsector0 checker reports path is down
  153. May 10 05:10:49 hv011 multipathd: sdj: readsector0 checker reports path is down
  154. May 10 05:10:49 hv011 multipathd: sdr: readsector0 checker reports path is down
  155. May 10 05:10:49 hv011 multipathd: sdt: readsector0 checker reports path is down
  156. May 10 05:10:49 hv011 multipathd: sdu: readsector0 checker reports path is down
  157. May 10 05:10:49 hv011 multipathd: sdb: readsector0 checker reports path is down
  158. May 10 05:10:51 hv011 multipathd: sdf: readsector0 checker reports path is down
  159. May 10 05:10:52 hv011 iscsid: connection3:0 is operational after recovery (1 attempts)
  160. May 10 05:10:54 hv011 multipathd: sdl: readsector0 checker reports path is down
  161. May 10 05:10:54 hv011 multipathd: sdo: readsector0 checker reports path is down
  162. May 10 05:10:55 hv011 iscsid: connection2:0 is operational after recovery (1 attempts)
  163. May 10 05:10:55 hv011 multipathd: sde: readsector0 checker reports path is down
  164. May 10 05:10:55 hv011 multipathd: sdg: readsector0 checker reports path is down
  165. May 10 05:10:55 hv011 multipathd: sdj: readsector0 checker reports path is down
  166. May 10 05:10:55 hv011 multipathd: sdr: readsector0 checker reports path is down
  167. May 10 05:10:55 hv011 multipathd: sdt: readsector0 checker reports path is down
  168. May 10 05:10:55 hv011 multipathd: sdu: readsector0 checker reports path is down
  169. May 10 05:10:55 hv011 multipathd: sdb: readsector0 checker reports path is down
  170. May 10 05:10:57 hv011 multipathd: sdf: readsector0 checker reports path is down
  171. May 10 05:10:59 hv011 iscsid: connection1:0 is operational after recovery (1 attempts)
  172. May 10 05:11:00 hv011 multipathd: sdl: readsector0 checker reports path is down
  173. May 10 05:11:00 hv011 multipathd: sdo: readsector0 checker reports path is down
  174. May 10 05:11:02 hv011 multipathd: sde: readsector0 checker reports path is down
  175. May 10 05:11:02 hv011 multipathd: sdg: readsector0 checker reports path is down
  176. May 10 05:11:02 hv011 multipathd: sdj: readsector0 checker reports path is down
  177. May 10 05:11:02 hv011 multipathd: sdr: readsector0 checker reports path is down
  178. May 10 05:11:02 hv011 multipathd: sdt: readsector0 checker reports path is down
  179. May 10 05:11:02 hv011 multipathd: sdu: readsector0 checker reports path is down
  180. May 10 05:11:13 hv011 multipathd: sdb: readsector0 checker reports path is up
  181. May 10 05:11:13 hv011 multipathd: 8:16: reinstated
  182. May 10 05:11:13 hv011 multipathd: mpath1: remaining active paths: 1
  183. May 10 05:11:13 hv011 multipathd: dm-2: add map (uevent)
  184. May 10 05:11:14 hv011 iscsid: connection4:0 is operational after recovery (2 attempts)
  185. May 10 05:11:15 hv011 multipathd: dm-2: devmap already registered
  186. May 10 05:11:15 hv011 multipathd: sdf: readsector0 checker reports path is up
  187. May 10 05:11:15 hv011 multipathd: 8:80: reinstated
  188. May 10 05:11:15 hv011 multipathd: mpath3: remaining active paths: 1
  189. May 10 05:11:15 hv011 multipathd: dm-4: add map (uevent)
  190. May 10 05:11:15 hv011 multipathd: dm-4: devmap already registered
  191. May 10 05:11:15 hv011 multipathd: 8:112: mark as failed
  192. May 10 05:11:17 hv011 multipathd: sdl: readsector0 checker reports path is up
  193. May 10 05:11:17 hv011 multipathd: 8:176: reinstated
  194. May 10 05:11:17 hv011 multipathd: mpath6: remaining active paths: 1
  195. May 10 05:11:17 hv011 multipathd: sdo: readsector0 checker reports path is up
  196. May 10 05:11:17 hv011 multipathd: 8:224: reinstated
  197. May 10 05:11:17 hv011 multipathd: mpath7: remaining active paths: 1
  198. May 10 05:11:17 hv011 multipathd: 8:192: mark as failed
  199. May 10 05:11:17 hv011 multipathd: dm-7: add map (uevent)
  200. May 10 05:11:17 hv011 multipathd: dm-7: devmap already registered
  201. May 10 05:11:17 hv011 multipathd: dm-8: add map (uevent)
  202. May 10 05:11:17 hv011 multipathd: dm-8: devmap already registered
  203. May 10 05:11:17 hv011 multipathd: 8:208: mark as failed
  204. May 10 05:11:20 hv011 multipathd: sde: readsector0 checker reports path is up
  205. May 10 05:11:20 hv011 multipathd: 8:64: reinstated
  206. May 10 05:11:20 hv011 multipathd: mpath2: remaining active paths: 1
  207. May 10 05:11:20 hv011 multipathd: sdg: readsector0 checker reports path is up
  208. May 10 05:11:20 hv011 multipathd: 8:96: reinstated
  209. May 10 05:11:20 hv011 multipathd: mpath4: remaining active paths: 1
  210. May 10 05:11:20 hv011 multipathd: sdj: readsector0 checker reports path is up
  211. May 10 05:11:20 hv011 multipathd: 8:144: reinstated
  212. May 10 05:11:20 hv011 multipathd: mpath5: remaining active paths: 1
  213. May 10 05:11:20 hv011 multipathd: sdr: readsector0 checker reports path is up
  214. May 10 05:11:20 hv011 multipathd: 65:16: reinstated
  215. May 10 05:11:20 hv011 multipathd: mpath8: remaining active paths: 1
  216. May 10 05:11:20 hv011 multipathd: sdt: readsector0 checker reports path is up
  217. May 10 05:11:20 hv011 multipathd: 65:48: reinstated
  218. May 10 05:11:20 hv011 multipathd: mpath9: remaining active paths: 1
  219. May 10 05:11:20 hv011 multipathd: sdu: readsector0 checker reports path is up
  220. May 10 05:11:20 hv011 multipathd: 65:64: reinstated
  221. May 10 05:11:20 hv011 multipathd: mpath10: remaining active paths: 1
  222. May 10 05:11:20 hv011 multipathd: 8:128: mark as failed
  223. May 10 05:11:20 hv011 multipathd: 8:160: mark as failed
  224. May 10 05:11:20 hv011 multipathd: 65:0: mark as failed
  225. May 10 05:11:20 hv011 multipathd: dm-3: add map (uevent)
  226. May 10 05:11:20 hv011 multipathd: dm-3: devmap already registered
  227. May 10 05:11:20 hv011 multipathd: dm-5: add map (uevent)
  228. May 10 05:11:20 hv011 multipathd: dm-5: devmap already registered
  229. May 10 05:11:20 hv011 multipathd: dm-6: add map (uevent)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement