API tools faq
paste
Guest User

hv004-02

a guest
Jun 12th, 2012
152
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 67.88 KB | None | 0 0
raw download clone embed print report
  1. Jun 1 22:35:55 hv004 kernel: connection1:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5738476340, last ping 5738481654, now 5738484234
  2. Jun 1 22:35:55 hv004 kernel: connection1:0: detected conn error (1011)
  3. Jun 1 22:35:56 hv004 kernel: device-mapper: multipath: Failing path 8:16.
  4. Jun 1 22:35:56 hv004 multipathd: sdb: readsector0 checker reports path is down
  5. Jun 1 22:35:56 hv004 multipathd: checker failed path 8:16 in map mpath4
  6. Jun 1 22:35:56 hv004 multipathd: mpath4: remaining active paths: 0
  7. Jun 1 22:35:56 hv004 kernel: connection4:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5738479375, last ping 5738480648, now 5738484438
  8. Jun 1 22:35:56 hv004 kernel: connection4:0: detected conn error (1011)
  9. Jun 1 22:36:04 hv004 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  10. Jun 1 22:36:04 hv004 iscsid: Kernel reported iSCSI connection 4:0 error (1011) state (3)
  11. Jun 1 22:36:07 hv004 kernel: connection2:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5738481380, last ping 5738484541, now 5738487197
  12. Jun 1 22:36:07 hv004 kernel: connection2:0: detected conn error (1011)
  13. Jun 1 22:36:15 hv004 kernel: session1: session recovery timed out after 15 secs
  14. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  15. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 14373513637
  16. Jun 1 22:36:15 hv004 kernel: device-mapper: multipath: Failing path 8:96.
  17. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  18. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8769344240
  19. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  20. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8765936560
  21. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  22. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8762593472
  23. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  24. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8738410992
  25. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  26. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8813646168
  27. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  28. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8738410760
  29. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  30. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8813646296
  31. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  32. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 2069640120
  33. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  34. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8738410920
  35. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  36. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8834182152
  37. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  38. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8813646488
  39. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  40. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8738410880
  41. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  42. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8813646088
  43. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  44. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8834182248
  45. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  46. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 2074760608
  47. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  48. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 2026133080
  49. Jun 1 22:36:15 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  50. Jun 1 22:36:15 hv004 kernel: end_request: I/O error, dev sdg, sector 8762790288
  51. Jun 1 22:36:15 hv004 kernel: device-mapper: multipath: Failing path 8:64.
  52. Jun 1 22:36:15 hv004 multipathd: sde: readsector0 checker reports path is down
  53. Jun 1 22:36:15 hv004 multipathd: checker failed path 8:64 in map mpath6
  54. Jun 1 22:36:15 hv004 multipathd: mpath6: remaining active paths: 0
  55. Jun 1 22:36:17 hv004 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  56. Jun 1 22:36:28 hv004 kernel: BUG: soft lockup - CPU#0 stuck for 60s! [swapper:0]
  57. Jun 1 22:36:28 hv004 kernel: CPU 0:
  58. Jun 1 22:36:28 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  59. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  60. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  61. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  62. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  63. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  64. Jun 1 22:36:28 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  65. Jun 1 22:36:28 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  66. Jun 1 22:36:28 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  67. Jun 1 22:36:28 hv004 kernel: RSP: e02b:ffffffff80797ee8 EFLAGS: 00000287
  68. Jun 1 22:36:28 hv004 kernel: RAX: ffffffff80751fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  69. Jun 1 22:36:28 hv004 kernel: RDX: ffffffffff578000 RSI: ffff880009287ea8 RDI: ffffffff888ce660
  70. Jun 1 22:36:28 hv004 kernel: RBP: ffffffff80851280 R08: ffff88006a600818 R09: 0000000000000000
  71. Jun 1 22:36:28 hv004 kernel: R10: ffff880009287ea8 R11: ffff880009287ec0 R12: ffff88005c61c7a0
  72. Jun 1 22:36:28 hv004 kernel: R13: 0000000000000100 R14: ffff88005c61c788 R15: ffffffff888c20fd
  73. Jun 1 22:36:28 hv004 kernel: FS: 00002b7613501be0(0000) GS:ffffffff80630000(0000) knlGS:0000000000000000
  74. Jun 1 22:36:28 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  75. Jun 1 22:36:28 hv004 kernel:
  76. Jun 1 22:36:28 hv004 kernel: Call Trace:
  77. Jun 1 22:36:28 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  78. Jun 1 22:36:28 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  79. Jun 1 22:36:28 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  80. Jun 1 22:36:28 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  81. Jun 1 22:36:28 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  82. Jun 1 22:36:28 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  83. Jun 1 22:36:28 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  84. Jun 1 22:36:28 hv004 kernel: <EOI> [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
  85. Jun 1 22:36:28 hv004 kernel: [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
  86. Jun 1 22:36:28 hv004 kernel: [<ffffffff803b6f22>] force_evtchn_callback+0xa/0xb
  87. Jun 1 22:36:28 hv004 kernel: [<ffffffff80261dc7>] thread_return+0x5e/0x113
  88. Jun 1 22:36:28 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  89. Jun 1 22:36:28 hv004 kernel: [<ffffffff8024ac8e>] cpu_idle+0xb8/0xba
  90. Jun 1 22:36:28 hv004 kernel: [<ffffffff8075ab11>] start_kernel+0x21f/0x224
  91. Jun 1 22:36:28 hv004 kernel: [<ffffffff8075a1e5>] _sinittext+0x1e5/0x1eb
  92. Jun 1 22:36:28 hv004 kernel:
  93. Jun 1 22:36:30 hv004 kernel: session4: session recovery timed out after 15 secs
  94. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:3: SCSI error: return code = 0x000f0000
  95. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdq, sector 152714456
  96. Jun 1 22:36:30 hv004 kernel: device-mapper: multipath: Failing path 65:0.
  97. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:3: SCSI error: return code = 0x000f0000
  98. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdq, sector 64692287
  99. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:3: SCSI error: return code = 0x000f0000
  100. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdq, sector 70039504
  101. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  102. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdr, sector 1541795096
  103. Jun 1 22:36:30 hv004 kernel: device-mapper: multipath: Failing path 65:16.
  104. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  105. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdr, sector 2711993695
  106. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  107. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdr, sector 1359536760
  108. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  109. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sds, sector 4130351663
  110. Jun 1 22:36:30 hv004 multipathd: sdl: readsector0 checker reports path is down
  111. Jun 1 22:36:30 hv004 kernel: device-mapper: multipath: Failing path 8:176.
  112. Jun 1 22:36:30 hv004 multipathd: checker failed path 8:176 in map mpath7
  113. Jun 1 22:36:30 hv004 kernel: device-mapper: multipath: Failing path 65:32.
  114. Jun 1 22:36:30 hv004 multipathd: mpath7: remaining active paths: 0
  115. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  116. Jun 1 22:36:30 hv004 multipathd: dm-3: add map (uevent)
  117. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sds, sector 4121590967
  118. Jun 1 22:36:30 hv004 multipathd: dm-3: devmap already registered
  119. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  120. Jun 1 22:36:30 hv004 multipathd: dm-5: add map (uevent)
  121. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sds, sector 4130978631
  122. Jun 1 22:36:30 hv004 multipathd: dm-5: devmap already registered
  123. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  124. Jun 1 22:36:30 hv004 multipathd: dm-2: add map (uevent)
  125. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sds, sector 1729214639
  126. Jun 1 22:36:30 hv004 multipathd: dm-2: devmap already registered
  127. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:6: SCSI error: return code = 0x000f0000
  128. Jun 1 22:36:30 hv004 multipathd: dm-7: add map (uevent)
  129. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdt, sector 1667076344
  130. Jun 1 22:36:30 hv004 multipathd: dm-7: devmap already registered
  131. Jun 1 22:36:30 hv004 kernel: device-mapper: multipath: Failing path 65:48.
  132. Jun 1 22:36:30 hv004 multipathd: dm-4: add map (uevent)
  133. Jun 1 22:36:30 hv004 kernel: sd 8:0:0:6: SCSI error: return code = 0x000f0000
  134. Jun 1 22:36:30 hv004 multipathd: dm-4: devmap already registered
  135. Jun 1 22:36:30 hv004 kernel: end_request: I/O error, dev sdt, sector 1665264400
  136. Jun 1 22:36:30 hv004 multipathd: dm-8: add map (uevent)
  137. Jun 1 22:36:30 hv004 multipathd: dm-8: devmap already registered
  138. Jun 1 22:36:30 hv004 multipathd: 65:0: mark as failed
  139. Jun 1 22:36:30 hv004 multipathd: mpath9: remaining active paths: 0
  140. Jun 1 22:36:30 hv004 multipathd: dm-9: add map (uevent)
  141. Jun 1 22:36:30 hv004 multipathd: dm-9: devmap already registered
  142. Jun 1 22:36:30 hv004 multipathd: dm-10: add map (uevent)
  143. Jun 1 22:36:30 hv004 multipathd: dm-10: devmap already registered
  144. Jun 1 22:36:30 hv004 multipathd: 65:32: mark as failed
  145. Jun 1 22:36:30 hv004 multipathd: mpath11: remaining active paths: 0
  146. Jun 1 22:36:30 hv004 multipathd: 65:48: mark as failed
  147. Jun 1 22:36:30 hv004 multipathd: mpath12: remaining active paths: 0
  148. Jun 1 22:36:30 hv004 multipathd: 8:96: mark as failed
  149. Jun 1 22:36:30 hv004 multipathd: mpath5: remaining active paths: 0
  150. Jun 1 22:36:31 hv004 multipathd: 65:16: mark as failed
  151. Jun 1 22:36:31 hv004 multipathd: mpath10: remaining active paths: 0
  152. Jun 1 22:36:35 hv004 multipathd: 8:112: mark as failed
  153. Jun 1 22:36:35 hv004 kernel: session2: session recovery timed out after 15 secs
  154. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  155. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834181704
  156. Jun 1 22:36:35 hv004 kernel: device-mapper: multipath: Failing path 8:112.
  157. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  158. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834182336
  159. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  160. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410784
  161. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  162. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2029839296
  163. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  164. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8765936328
  165. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  166. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2025920272
  167. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  168. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2020183096
  169. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  170. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646240
  171. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  172. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8762528192
  173. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  174. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410880
  175. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  176. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410760
  177. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  178. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646488
  179. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  180. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410992
  181. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  182. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646296
  183. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  184. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646168
  185. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  186. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646088
  187. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  188. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8769344240
  189. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  190. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8765936560
  191. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  192. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834182064
  193. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  194. Jun 1 22:36:35 hv004 multipathd: dm-5: add map (uevent)
  195. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834181880
  196. Jun 1 22:36:35 hv004 multipathd: dm-5: devmap already registered
  197. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  198. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410984
  199. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  200. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8730546336
  201. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  202. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646568
  203. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  204. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646152
  205. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  206. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8765936912
  207. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  208. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813645976
  209. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  210. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410888
  211. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  212. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8762602808
  213. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  214. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8813646392
  215. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  216. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8762790288
  217. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  218. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8762593472
  219. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  220. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 14373513637
  221. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  222. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8738410920
  223. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  224. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834182152
  225. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  226. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2026133080
  227. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  228. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2069640120
  229. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  230. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 2074760608
  231. Jun 1 22:36:35 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  232. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdh, sector 8834182248
  233. Jun 1 22:36:35 hv004 kernel: sd 8:0:0:2: SCSI error: return code = 0x000f0000
  234. Jun 1 22:36:35 hv004 kernel: end_request: I/O error, dev sdp, sector 588295859
  235. Jun 1 22:36:35 hv004 kernel: device-mapper: multipath: Failing path 8:240.
  236. Jun 1 22:36:35 hv004 multipathd: 8:240: mark as failed
  237. Jun 1 22:36:35 hv004 multipathd: mpath8: remaining active paths: 0
  238. Jun 1 22:36:35 hv004 multipathd: sdg: readsector0 checker reports path is down
  239. Jun 1 22:36:35 hv004 multipathd: sdp: readsector0 checker reports path is down
  240. Jun 1 22:36:35 hv004 multipathd: sdq: readsector0 checker reports path is down
  241. Jun 1 22:36:35 hv004 multipathd: sdr: readsector0 checker reports path is down
  242. Jun 1 22:36:35 hv004 multipathd: sds: readsector0 checker reports path is down
  243. Jun 1 22:36:35 hv004 multipathd: sdt: readsector0 checker reports path is down
  244. Jun 1 22:36:35 hv004 multipathd: sdu: readsector0 checker reports path is down
  245. Jun 1 22:36:35 hv004 multipathd: checker failed path 65:64 in map mpath13
  246. Jun 1 22:36:35 hv004 kernel: device-mapper: multipath: Failing path 65:64.
  247. Jun 1 22:36:35 hv004 multipathd: mpath13: remaining active paths: 0
  248. Jun 1 22:36:36 hv004 multipathd: dm-6: add map (uevent)
  249. Jun 1 22:36:36 hv004 multipathd: dm-6: devmap already registered
  250. Jun 1 22:36:36 hv004 multipathd: dm-11: add map (uevent)
  251. Jun 1 22:36:36 hv004 multipathd: dm-11: devmap already registered
  252. Jun 1 22:36:37 hv004 nrpe[9601]: Error: Could not complete SSL handshake. 5
  253. Jun 1 22:36:37 hv004 multipathd: sdb: readsector0 checker reports path is down
  254. Jun 1 22:36:37 hv004 multipathd: sde: readsector0 checker reports path is down
  255. Jun 1 22:36:37 hv004 multipathd: sdl: readsector0 checker reports path is down
  256. Jun 1 22:36:41 hv004 iscsid: connection4:0 is operational after recovery (1 attempts)
  257. Jun 1 22:36:41 hv004 kernel: BUG: soft lockup - CPU#2 stuck for 60s! [events/2:76]
  258. Jun 1 22:36:41 hv004 kernel: CPU 2:
  259. Jun 1 22:36:41 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  260. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  261. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  262. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  263. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  264. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  265. Jun 1 22:36:41 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  266. Jun 1 22:36:41 hv004 kernel: Pid: 76, comm: events/2 Not tainted 2.6.18-274.7.1.el5xen #1
  267. Jun 1 22:36:41 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  268. Jun 1 22:36:41 hv004 kernel: RSP: e02b:ffff88007ff13ee8 EFLAGS: 00000287
  269. Jun 1 22:36:41 hv004 kernel: RAX: ffff88007f80dfd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  270. Jun 1 22:36:41 hv004 kernel: RDX: ffffffffff578080 RSI: ffff88004c070658 RDI: ffffffff888ce660
  271. Jun 1 22:36:41 hv004 kernel: RBP: ffff88007ff04000 R08: ffff880038b37818 R09: 0000000000000002
  272. Jun 1 22:36:41 hv004 kernel: R10: ffff88004c070658 R11: ffff88004c070670 R12: ffff8800389eb410
  273. Jun 1 22:36:41 hv004 kernel: R13: 0000000000000100 R14: ffff8800389eb3f8 R15: ffffffff888c20fd
  274. Jun 1 22:36:41 hv004 kernel: FS: 00002b2912dd8af0(0000) GS:ffffffff80630100(0000) knlGS:0000000000000000
  275. Jun 1 22:36:41 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  276. Jun 1 22:36:41 hv004 kernel:
  277. Jun 1 22:36:41 hv004 kernel: Call Trace:
  278. Jun 1 22:36:41 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  279. Jun 1 22:36:41 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  280. Jun 1 22:36:41 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  281. Jun 1 22:36:41 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  282. Jun 1 22:36:41 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  283. Jun 1 22:36:41 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  284. Jun 1 22:36:41 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  285. Jun 1 22:36:41 hv004 kernel: <EOI> [<ffffffff802d2528>] drain_array+0x28/0xd3
  286. Jun 1 22:36:41 hv004 kernel: [<ffffffff802d2f6e>] cache_reap+0x0/0x128
  287. Jun 1 22:36:41 hv004 kernel: [<ffffffff802d2fcb>] cache_reap+0x5d/0x128
  288. Jun 1 22:36:41 hv004 kernel: [<ffffffff8024f152>] run_workqueue+0x9e/0xfb
  289. Jun 1 22:36:41 hv004 kernel: [<ffffffff8024b9df>] worker_thread+0x0/0x122
  290. Jun 1 22:36:41 hv004 kernel: [<ffffffff8024bacf>] worker_thread+0xf0/0x122
  291. Jun 1 22:36:41 hv004 kernel: [<ffffffff802895dd>] default_wake_function+0x0/0xe
  292. Jun 1 22:36:41 hv004 kernel: [<ffffffff8023398b>] kthread+0xfe/0x132
  293. Jun 1 22:36:41 hv004 kernel: [<ffffffff8025fb2c>] child_rip+0xa/0x12
  294. Jun 1 22:36:41 hv004 kernel: [<ffffffff8023388d>] kthread+0x0/0x132
  295. Jun 1 22:36:41 hv004 kernel: [<ffffffff8025fb22>] child_rip+0x0/0x12
  296. Jun 1 22:36:41 hv004 kernel:
  297. Jun 1 22:36:50 hv004 multipathd: sdg: readsector0 checker reports path is down
  298. Jun 1 22:36:50 hv004 iscsid: connection2:0 is operational after recovery (1 attempts)
  299. Jun 1 22:36:54 hv004 kernel: BUG: soft lockup - CPU#9 stuck for 60s! [swapper:0]
  300. Jun 1 22:36:54 hv004 kernel: CPU 9:
  301. Jun 1 22:36:54 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  302. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  303. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  304. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  305. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  306. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  307. Jun 1 22:36:54 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  308. Jun 1 22:36:54 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  309. Jun 1 22:36:54 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  310. Jun 1 22:36:54 hv004 kernel: RSP: e02b:ffff8800015638e8 EFLAGS: 00000287
  311. Jun 1 22:36:54 hv004 kernel: RAX: ffff880007bd9fd8 RBX: ffffffff888ce660 RCX: 00000000cd5879ae
  312. Jun 1 22:36:54 hv004 kernel: RDX: 000000000000136b RSI: 0000000000002000 RDI: ffffffff888ce660
  313. Jun 1 22:36:54 hv004 kernel: RBP: ffff8800589dfc48 R08: 00000000554f8f06 R09: ffffffff888cd960
  314. Jun 1 22:36:54 hv004 kernel: R10: 0000000000000002 R11: 0000000000000001 R12: 00000000000001db
  315. Jun 1 22:36:54 hv004 kernel: R13: ffff880001563a10 R14: ffff880001563a10 R15: ffffffff808d7280
  316. Jun 1 22:36:59 hv004 kernel: FS: 00002afeba786170(0000) GS:ffffffff80630480(0000) knlGS:0000000000000000
  317. Jun 1 22:36:59 hv004 kernel: CS: e033 DS: 002b ES: 002b
  318. Jun 1 22:36:59 hv004 kernel:
  319. Jun 1 22:36:59 hv004 kernel: Call Trace:
  320. Jun 1 22:36:59 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  321. Jun 1 22:36:59 hv004 kernel: [<ffffffff888c2f0e>] :ip_conntrack:__ip_conntrack_confirm+0x56/0x2f8
  322. Jun 1 22:36:59 hv004 kernel: [<ffffffff888c115a>] :ip_conntrack:ip_confirm+0x29/0x39
  323. Jun 1 22:36:59 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  324. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875e8e3>] :bridge:br_nf_dev_queue_xmit+0x0/0x3b
  325. Jun 1 22:36:59 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  326. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875e8e3>] :bridge:br_nf_dev_queue_xmit+0x0/0x3b
  327. Jun 1 22:36:59 hv004 kernel: [<ffffffff88777bd0>] :ip_tables:ipt_do_table+0x2ca/0x2fa
  328. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ec1f>] :bridge:br_nf_post_routing+0x167/0x197
  329. Jun 1 22:36:59 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  330. Jun 1 22:36:59 hv004 kernel: [<ffffffff88759f95>] :bridge:br_dev_queue_push_xmit+0x0/0x200
  331. Jun 1 22:36:59 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  332. Jun 1 22:36:59 hv004 kernel: [<ffffffff88759f95>] :bridge:br_dev_queue_push_xmit+0x0/0x200
  333. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875a1d4>] :bridge:br_forward_finish+0x3f/0x51
  334. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875eab0>] :bridge:br_nf_forward_finish+0xf7/0xff
  335. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875f34e>] :bridge:br_nf_forward_ip+0x150/0x160
  336. Jun 1 22:36:59 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  337. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  338. Jun 1 22:36:59 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  339. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  340. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875a23e>] :bridge:__br_forward+0x58/0x9c
  341. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ae69>] :bridge:br_handle_frame_finish+0x128/0x1d3
  342. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875f123>] :bridge:br_nf_pre_routing_finish+0x2e9/0x2f8
  343. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  344. Jun 1 22:36:59 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  345. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  346. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875fd20>] :bridge:br_nf_pre_routing+0x600/0x61c
  347. Jun 1 22:36:59 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  348. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  349. Jun 1 22:36:59 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  350. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  351. Jun 1 22:36:59 hv004 kernel: [<ffffffff8875b082>] :bridge:br_handle_frame+0x16e/0x1a4
  352. Jun 1 22:36:59 hv004 kernel: [<ffffffff802214c5>] netif_receive_skb+0x3a8/0x4c4
  353. Jun 1 22:36:59 hv004 kernel: [<ffffffff88257d01>] :igb:igb_poll+0x72b/0xb55
  354. Jun 1 22:36:59 hv004 kernel: [<ffffffff8026fc98>] timer_interrupt+0x3d2/0x3e6
  355. Jun 1 22:36:59 hv004 kernel: [<ffffffff803b72dd>] unmask_evtchn+0x2d/0xd9
  356. Jun 1 22:36:59 hv004 kernel: [<ffffffff8020d09d>] net_rx_action+0xb4/0x1c6
  357. Jun 1 22:36:59 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  358. Jun 1 22:36:59 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  359. Jun 1 22:36:59 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  360. Jun 1 22:36:59 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  361. Jun 1 22:36:59 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  362. Jun 1 22:36:59 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  363. Jun 1 22:36:59 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  364. Jun 1 22:36:59 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  365. Jun 1 22:36:59 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  366. Jun 1 22:36:59 hv004 kernel:
  367. Jun 1 22:37:00 hv004 multipathd: sdp: readsector0 checker reports path is up
  368. Jun 1 22:37:00 hv004 multipathd: 8:240: reinstated
  369. Jun 1 22:37:00 hv004 multipathd: mpath8: remaining active paths: 1
  370. Jun 1 22:37:01 hv004 multipathd: sdq: readsector0 checker reports path is up
  371. Jun 1 22:37:01 hv004 multipathd: 65:0: reinstated
  372. Jun 1 22:37:01 hv004 multipathd: mpath9: remaining active paths: 1
  373. Jun 1 22:37:10 hv004 multipathd: sdr: readsector0 checker reports path is up
  374. Jun 1 22:37:10 hv004 multipathd: 65:16: reinstated
  375. Jun 1 22:37:10 hv004 multipathd: mpath10: remaining active paths: 1
  376. Jun 1 22:37:16 hv004 multipathd: sds: readsector0 checker reports path is up
  377. Jun 1 22:37:16 hv004 multipathd: 65:32: reinstated
  378. Jun 1 22:37:16 hv004 multipathd: mpath11: remaining active paths: 1
  379. Jun 1 22:37:28 hv004 kernel: BUG: soft lockup - CPU#0 stuck for 60s! [swapper:0]
  380. Jun 1 22:37:28 hv004 kernel: CPU 0:
  381. Jun 1 22:37:28 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  382. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  383. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  384. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  385. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  386. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  387. Jun 1 22:37:28 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  388. Jun 1 22:37:28 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  389. Jun 1 22:37:28 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  390. Jun 1 22:37:28 hv004 kernel: RSP: e02b:ffffffff80797ee8 EFLAGS: 00000287
  391. Jun 1 22:37:28 hv004 kernel: RAX: ffffffff80751fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  392. Jun 1 22:37:28 hv004 kernel: RDX: ffffffffff578000 RSI: ffff88002118e788 RDI: ffffffff888ce660
  393. Jun 1 22:37:28 hv004 kernel: RBP: ffffffff80851280 R08: ffff88006a600818 R09: 0000000000000000
  394. Jun 1 22:37:28 hv004 kernel: R10: ffff88002118e788 R11: ffff88002118e7a0 R12: ffff88001de3cd90
  395. Jun 1 22:37:28 hv004 kernel: R13: 0000000000000100 R14: ffff88001de3cd78 R15: ffffffff888c20fd
  396. Jun 1 22:37:28 hv004 kernel: FS: 00002b7613501be0(0000) GS:ffffffff80630000(0000) knlGS:0000000000000000
  397. Jun 1 22:38:05 hv004 nrpe[9680]: Could not read request from client, bailing out...
  398. Jun 1 22:38:15 hv004 multipathd: sdt: readsector0 checker reports path is up
  399. Jun 1 22:38:15 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  400. Jun 1 22:38:15 hv004 kernel:
  401. Jun 1 22:38:15 hv004 kernel: Call Trace:
  402. Jun 1 22:38:15 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  403. Jun 1 22:38:15 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  404. Jun 1 22:38:15 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  405. Jun 1 22:38:15 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  406. Jun 1 22:38:15 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  407. Jun 1 22:38:15 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  408. Jun 1 22:38:15 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  409. Jun 1 22:38:15 hv004 kernel: <EOI> [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
  410. Jun 1 22:38:15 hv004 kernel: [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
  411. Jun 1 22:38:15 hv004 kernel: [<ffffffff803b6f22>] force_evtchn_callback+0xa/0xb
  412. Jun 1 22:38:15 hv004 kernel: [<ffffffff80261dc7>] thread_return+0x5e/0x113
  413. Jun 1 22:38:15 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  414. Jun 1 22:38:15 hv004 kernel: [<ffffffff8024ac8e>] cpu_idle+0xb8/0xba
  415. Jun 1 22:38:15 hv004 kernel: [<ffffffff8075ab11>] start_kernel+0x21f/0x224
  416. Jun 1 22:38:15 hv004 kernel: [<ffffffff8075a1e5>] _sinittext+0x1e5/0x1eb
  417. Jun 1 22:38:15 hv004 kernel:
  418. Jun 1 22:38:15 hv004 kernel: BUG: soft lockup - CPU#2 stuck for 60s! [events/2:76]
  419. Jun 1 22:38:15 hv004 kernel: CPU 2:
  420. Jun 1 22:38:15 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  421. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  422. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  423. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  424. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  425. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  426. Jun 1 22:38:15 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  427. Jun 1 22:38:15 hv004 kernel: Pid: 76, comm: events/2 Not tainted 2.6.18-274.7.1.el5xen #1
  428. Jun 1 22:38:15 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  429. Jun 1 22:38:15 hv004 kernel: RSP: e02b:ffff88007ff13ee8 EFLAGS: 00000287
  430. Jun 1 22:38:15 hv004 kernel: RAX: ffff88007f80dfd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  431. Jun 1 22:38:15 hv004 kernel: RDX: ffffffffff578080 RSI: ffff880038b379c8 RDI: ffffffff888ce660
  432. Jun 1 22:38:16 hv004 kernel: RBP: ffff88007ff04000 R08: ffff880038b37818 R09: 0000000000000002
  433. Jun 1 22:38:16 hv004 kernel: R10: ffff880054403788 R11: ffff8800544037a0 R12: ffff880039c338d0
  434. Jun 1 22:38:16 hv004 kernel: R13: 0000000000000100 R14: ffff880039c338b8 R15: ffffffff888c20fd
  435. Jun 1 22:38:16 hv004 kernel: FS: 00002b2912dd8af0(0000) GS:ffffffff80630100(0000) knlGS:0000000000000000
  436. Jun 1 22:38:16 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  437. Jun 1 22:38:16 hv004 kernel:
  438. Jun 1 22:38:16 hv004 kernel: Call Trace:
  439. Jun 1 22:38:16 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  440. Jun 1 22:38:16 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  441. Jun 1 22:38:16 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  442. Jun 1 22:38:16 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  443. Jun 1 22:38:16 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  444. Jun 1 22:38:16 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  445. Jun 1 22:38:16 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  446. Jun 1 22:38:16 hv004 kernel: <EOI> [<ffffffff802d2528>] drain_array+0x28/0xd3
  447. Jun 1 22:38:16 hv004 kernel: [<ffffffff802d2f6e>] cache_reap+0x0/0x128
  448. Jun 1 22:38:16 hv004 kernel: [<ffffffff802d2fcb>] cache_reap+0x5d/0x128
  449. Jun 1 22:38:16 hv004 kernel: [<ffffffff8024f152>] run_workqueue+0x9e/0xfb
  450. Jun 1 22:38:16 hv004 kernel: [<ffffffff8024b9df>] worker_thread+0x0/0x122
  451. Jun 1 22:38:16 hv004 kernel: [<ffffffff8024bacf>] worker_thread+0xf0/0x122
  452. Jun 1 22:38:16 hv004 kernel: [<ffffffff802895dd>] default_wake_function+0x0/0xe
  453. Jun 1 22:38:16 hv004 kernel: [<ffffffff8023398b>] kthread+0xfe/0x132
  454. Jun 1 22:38:16 hv004 kernel: [<ffffffff8025fb2c>] child_rip+0xa/0x12
  455. Jun 1 22:38:16 hv004 kernel: [<ffffffff8023388d>] kthread+0x0/0x132
  456. Jun 1 22:38:16 hv004 kernel: [<ffffffff8025fb22>] child_rip+0x0/0x12
  457. Jun 1 22:38:16 hv004 kernel:
  458. Jun 1 22:38:16 hv004 multipathd: 65:48: reinstated
  459. Jun 1 22:38:16 hv004 multipathd: mpath12: remaining active paths: 1
  460. Jun 1 22:38:16 hv004 multipathd: sdu: readsector0 checker reports path is up
  461. Jun 1 22:38:16 hv004 multipathd: 65:64: reinstated
  462. Jun 1 22:38:16 hv004 multipathd: mpath13: remaining active paths: 1
  463. Jun 1 22:38:16 hv004 multipathd: dm-6: add map (uevent)
  464. Jun 1 22:38:16 hv004 multipathd: dm-6: devmap already registered
  465. Jun 1 22:38:16 hv004 multipathd: dm-7: add map (uevent)
  466. Jun 1 22:38:16 hv004 multipathd: dm-7: devmap already registered
  467. Jun 1 22:38:16 hv004 multipathd: dm-8: add map (uevent)
  468. Jun 1 22:38:16 hv004 multipathd: dm-8: devmap already registered
  469. Jun 1 22:38:16 hv004 multipathd: dm-9: add map (uevent)
  470. Jun 1 22:38:16 hv004 multipathd: dm-9: devmap already registered
  471. Jun 1 22:38:16 hv004 multipathd: dm-10: add map (uevent)
  472. Jun 1 22:38:16 hv004 multipathd: dm-10: devmap already registered
  473. Jun 1 22:38:16 hv004 multipathd: dm-11: add map (uevent)
  474. Jun 1 22:38:16 hv004 multipathd: dm-11: devmap already registered
  475. Jun 1 22:38:16 hv004 multipathd: sdb: readsector0 checker reports path is up
  476. Jun 1 22:38:16 hv004 multipathd: 8:16: reinstated
  477. Jun 1 22:38:16 hv004 multipathd: mpath4: remaining active paths: 1
  478. Jun 1 22:38:16 hv004 multipathd: sde: readsector0 checker reports path is up
  479. Jun 1 22:38:16 hv004 multipathd: 8:64: reinstated
  480. Jun 1 22:38:16 hv004 multipathd: mpath6: remaining active paths: 1
  481. Jun 1 22:38:16 hv004 multipathd: sdl: readsector0 checker reports path is up
  482. Jun 1 22:38:16 hv004 multipathd: 8:176: reinstated
  483. Jun 1 22:38:16 hv004 multipathd: mpath7: remaining active paths: 1
  484. Jun 1 22:38:16 hv004 multipathd: dm-3: add map (uevent)
  485. Jun 1 22:38:16 hv004 multipathd: dm-3: devmap already registered
  486. Jun 1 22:38:16 hv004 multipathd: dm-2: add map (uevent)
  487. Jun 1 22:38:16 hv004 multipathd: dm-2: devmap already registered
  488. Jun 1 22:38:16 hv004 multipathd: dm-4: add map (uevent)
  489. Jun 1 22:38:16 hv004 multipathd: dm-4: devmap already registered
  490. Jun 1 22:38:16 hv004 iscsid: connection1:0 is operational after recovery (3 attempts)
  491. Jun 1 22:38:20 hv004 multipathd: sdg: readsector0 checker reports path is up
  492. Jun 1 22:38:20 hv004 multipathd: 8:96: reinstated
  493. Jun 1 22:38:20 hv004 multipathd: mpath5: remaining active paths: 1
  494. Jun 1 22:38:21 hv004 multipathd: 8:112: mark as failed
  495. Jun 1 22:38:21 hv004 multipathd: dm-5: add map (uevent)
  496. Jun 1 22:38:21 hv004 multipathd: dm-5: devmap already registered
  497. Jun 1 22:45:30 hv004 kernel: connection4:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5738623390, last ping 5738624967, now 5738627939
  498. Jun 1 22:45:30 hv004 kernel: connection4:0: detected conn error (1011)
  499. Jun 1 22:45:31 hv004 iscsid: Kernel reported iSCSI connection 4:0 error (1011) state (3)
  500. Jun 1 22:45:49 hv004 kernel: session4: session recovery timed out after 15 secs
  501. Jun 1 22:45:49 hv004 kernel: sd 8:0:0:0: SCSI error: return code = 0x000f0000
  502. Jun 1 22:45:49 hv004 kernel: end_request: I/O error, dev sde, sector 46273528
  503. Jun 1 22:45:49 hv004 kernel: device-mapper: multipath: Failing path 8:64.
  504. Jun 1 22:45:49 hv004 kernel: sd 8:0:0:3: SCSI error: return code = 0x000f0000
  505. Jun 1 22:45:49 hv004 kernel: end_request: I/O error, dev sdq, sector 69283744
  506. Jun 1 22:45:49 hv004 kernel: device-mapper: multipath: Failing path 65:0.
  507. Jun 1 22:45:49 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  508. Jun 1 22:45:49 hv004 kernel: end_request: I/O error, dev sdr, sector 1384805056
  509. Jun 1 22:45:49 hv004 kernel: device-mapper: multipath: Failing path 65:16.
  510. Jun 1 22:45:49 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  511. Jun 1 22:45:49 hv004 kernel: end_request: I/O error, dev sdr, sector 1358928880
  512. Jun 1 22:45:49 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  513. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdr, sector 1603631024
  514. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  515. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdr, sector 1376191096
  516. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  517. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdr, sector 2670375280
  518. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:4: SCSI error: return code = 0x000f0000
  519. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdr, sector 1546747240
  520. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  521. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 4110832479
  522. Jun 1 22:46:01 hv004 kernel: device-mapper: multipath: Failing path 65:32.
  523. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  524. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 1709534135
  525. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  526. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 4041354759
  527. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  528. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 4039020559
  529. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  530. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 4032888119
  531. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:5: SCSI error: return code = 0x000f0000
  532. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sds, sector 4041248199
  533. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:6: SCSI error: return code = 0x000f0000
  534. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdt, sector 384862216
  535. Jun 1 22:46:01 hv004 kernel: device-mapper: multipath: Failing path 65:48.
  536. Jun 1 22:46:01 hv004 kernel: sd 8:0:0:6: SCSI error: return code = 0x000f0000
  537. Jun 1 22:46:01 hv004 kernel: end_request: I/O error, dev sdt, sector 1665264728
  538. Jun 1 22:46:03 hv004 kernel: device-mapper: multipath: Failing path 8:176.
  539. Jun 1 22:46:03 hv004 kernel: device-mapper: multipath: Failing path 8:240.
  540. Jun 1 22:46:03 hv004 kernel: device-mapper: multipath: Failing path 65:64.
  541. Jun 1 22:46:03 hv004 multipathd: sde: readsector0 checker reports path is down
  542. Jun 1 22:46:03 hv004 multipathd: checker failed path 8:64 in map mpath6
  543. Jun 1 22:46:03 hv004 multipathd: mpath6: remaining active paths: 0
  544. Jun 1 22:46:03 hv004 multipathd: sdl: readsector0 checker reports path is down
  545. Jun 1 22:46:03 hv004 multipathd: checker failed path 8:176 in map mpath7
  546. Jun 1 22:46:03 hv004 multipathd: mpath7: remaining active paths: 0
  547. Jun 1 22:46:03 hv004 multipathd: sdp: readsector0 checker reports path is down
  548. Jun 1 22:46:03 hv004 multipathd: checker failed path 8:240 in map mpath8
  549. Jun 1 22:46:03 hv004 multipathd: mpath8: remaining active paths: 0
  550. Jun 1 22:46:03 hv004 multipathd: sdq: readsector0 checker reports path is down
  551. Jun 1 22:46:03 hv004 multipathd: checker failed path 65:0 in map mpath9
  552. Jun 1 22:46:03 hv004 multipathd: mpath9: remaining active paths: 0
  553. Jun 1 22:46:03 hv004 multipathd: sdr: readsector0 checker reports path is down
  554. Jun 1 22:46:03 hv004 multipathd: checker failed path 65:16 in map mpath10
  555. Jun 1 22:46:03 hv004 multipathd: mpath10: remaining active paths: 0
  556. Jun 1 22:46:03 hv004 multipathd: sds: readsector0 checker reports path is down
  557. Jun 1 22:46:03 hv004 multipathd: checker failed path 65:32 in map mpath11
  558. Jun 1 22:46:03 hv004 multipathd: mpath11: remaining active paths: 0
  559. Jun 1 22:46:03 hv004 multipathd: sdt: readsector0 checker reports path is down
  560. Jun 1 22:46:03 hv004 multipathd: checker failed path 65:48 in map mpath12
  561. Jun 1 22:46:03 hv004 multipathd: mpath12: remaining active paths: 0
  562. Jun 1 22:46:03 hv004 multipathd: sdu: readsector0 checker reports path is down
  563. Jun 1 22:46:03 hv004 multipathd: checker failed path 65:64 in map mpath13
  564. Jun 1 22:46:03 hv004 multipathd: mpath13: remaining active paths: 0
  565. Jun 1 22:46:03 hv004 multipathd: dm-2: add map (uevent)
  566. Jun 1 22:46:03 hv004 multipathd: dm-2: devmap already registered
  567. Jun 1 22:46:03 hv004 multipathd: dm-7: add map (uevent)
  568. Jun 1 22:46:03 hv004 multipathd: dm-7: devmap already registered
  569. Jun 1 22:46:03 hv004 multipathd: dm-8: add map (uevent)
  570. Jun 1 22:46:03 hv004 multipathd: dm-8: devmap already registered
  571. Jun 1 22:46:03 hv004 multipathd: dm-9: add map (uevent)
  572. Jun 1 22:46:03 hv004 multipathd: dm-9: devmap already registered
  573. Jun 1 22:46:03 hv004 multipathd: dm-10: add map (uevent)
  574. Jun 1 22:46:03 hv004 multipathd: dm-10: devmap already registered
  575. Jun 1 22:46:03 hv004 multipathd: dm-4: add map (uevent)
  576. Jun 1 22:46:03 hv004 multipathd: dm-4: devmap already registered
  577. Jun 1 22:46:03 hv004 multipathd: dm-6: add map (uevent)
  578. Jun 1 22:46:03 hv004 multipathd: dm-6: devmap already registered
  579. Jun 1 22:46:03 hv004 multipathd: dm-11: add map (uevent)
  580. Jun 1 22:46:03 hv004 multipathd: dm-11: devmap already registered
  581. Jun 1 22:46:15 hv004 kernel: BUG: soft lockup - CPU#5 stuck for 60s! [events/5:79]
  582. Jun 1 22:46:15 hv004 kernel: CPU 5:
  583. Jun 1 22:46:15 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  584. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  585. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  586. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  587. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  588. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  589. Jun 1 22:46:15 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  590. Jun 1 22:46:15 hv004 kernel: Pid: 79, comm: events/5 Not tainted 2.6.18-274.7.1.el5xen #1
  591. Jun 1 22:46:15 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  592. Jun 1 22:46:15 hv004 kernel: RSP: e02b:ffff88007ff63ee8 EFLAGS: 00000287
  593. Jun 1 22:46:15 hv004 kernel: RAX: ffff88007f817fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  594. Jun 1 22:46:15 hv004 kernel: RDX: ffffffffff578140 RSI: ffff880037111528 RDI: ffffffff888ce660
  595. Jun 1 22:46:51 hv004 iscsid: connection4:0 is operational after recovery (3 attempts)
  596. Jun 1 22:47:17 hv004 multipathd: sde: readsector0 checker reports path is up
  597. Jun 1 22:47:32 hv004 kernel: RBP: ffff88007ff54000 R08: ffff880032e33c18 R09: 0000000000000005
  598. Jun 1 22:47:33 hv004 nrpe[10594]: Error: Could not complete SSL handshake. 5
  599. Jun 1 22:47:34 hv004 kernel: R10: ffff880037111528 R11: ffff880037111540 R12: ffff8800116691b0
  600. Jun 1 22:47:34 hv004 kernel: R13: 0000000000000100 R14: ffff880011669198 R15: ffffffff888c20fd
  601. Jun 1 22:47:34 hv004 kernel: FS: 00002b82dbd75f70(0000) GS:ffffffff80630280(0000) knlGS:0000000000000000
  602. Jun 1 22:47:34 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  603. Jun 1 22:47:34 hv004 kernel:
  604. Jun 1 22:47:34 hv004 kernel: Call Trace:
  605. Jun 1 22:47:34 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  606. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  607. Jun 1 22:47:34 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  608. Jun 1 22:47:34 hv004 kernel: [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  609. Jun 1 22:47:34 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  610. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  611. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  612. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  613. Jun 1 22:47:34 hv004 kernel: <EOI> [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  614. Jun 1 22:47:34 hv004 kernel: [<ffffffff88252b88>] :igb:igb_update_stats+0x42a/0x6c3
  615. Jun 1 22:47:34 hv004 kernel: [<ffffffff882582dc>] :igb:igb_watchdog_task+0x1b1/0x331
  616. Jun 1 22:47:34 hv004 kernel: [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  617. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024f152>] run_workqueue+0x9e/0xfb
  618. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024b9df>] worker_thread+0x0/0x122
  619. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024bacf>] worker_thread+0xf0/0x122
  620. Jun 1 22:47:34 hv004 kernel: [<ffffffff802895dd>] default_wake_function+0x0/0xe
  621. Jun 1 22:47:34 hv004 kernel: [<ffffffff8023398b>] kthread+0xfe/0x132
  622. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fb2c>] child_rip+0xa/0x12
  623. Jun 1 22:47:34 hv004 kernel: [<ffffffff8023388d>] kthread+0x0/0x132
  624. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fb22>] child_rip+0x0/0x12
  625. Jun 1 22:47:34 hv004 kernel:
  626. Jun 1 22:47:34 hv004 kernel: BUG: soft lockup - CPU#12 stuck for 60s! [swapper:0]
  627. Jun 1 22:47:34 hv004 kernel: CPU 12:
  628. Jun 1 22:47:34 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  629. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  630. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  631. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  632. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  633. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  634. Jun 1 22:47:34 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  635. Jun 1 22:47:34 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  636. Jun 1 22:47:34 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  637. Jun 1 22:47:34 hv004 kernel: RSP: e02b:ffff88007fe37a48 EFLAGS: 00000287
  638. Jun 1 22:47:34 hv004 kernel: RAX: ffff880007be3fd8 RBX: ffffffff888ce660 RCX: 0000000000007530
  639. Jun 1 22:47:34 hv004 kernel: RDX: ffff8800257655c0 RSI: 0000000000000002 RDI: ffffffff888ce660
  640. Jun 1 22:47:34 hv004 kernel: RBP: ffff880008c0ec48 R08: 0000000000000001 R09: ffffffff8876af70
  641. Jun 1 22:47:34 hv004 kernel: R10: 000000000000d200 R11: ffff880008c0ec48 R12: ffffffff80792f20
  642. Jun 1 22:47:34 hv004 kernel: R13: 0000000000007530 R14: 0000000000000001 R15: 0000000000000002
  643. Jun 1 22:47:34 hv004 kernel: FS: 00002b85f9885f50(0000) GS:ffffffff80630600(0000) knlGS:0000000000000000
  644. Jun 1 22:47:34 hv004 kernel: CS: e033 DS: 002b ES: 002b
  645. Jun 1 22:47:34 hv004 kernel:
  646. Jun 1 22:47:34 hv004 kernel: Call Trace:
  647. Jun 1 22:47:34 hv004 kernel: <IRQ> [<ffffffff88759f95>] :bridge:br_dev_queue_push_xmit+0x0/0x200
  648. Jun 1 22:47:34 hv004 kernel: [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  649. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c1dcf>] :ip_conntrack:__ip_ct_refresh_acct+0x2a/0x153
  650. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c48d5>] :ip_conntrack:tcp_packet+0xa3f/0xa7e
  651. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875a1d4>] :bridge:br_forward_finish+0x3f/0x51
  652. Jun 1 22:47:34 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  653. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  654. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c3e1e>] :ip_conntrack:tcp_error+0x129/0x1a1
  655. Jun 1 22:47:34 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  656. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  657. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c1869>] :ip_conntrack:__ip_conntrack_find+0xd/0xb7
  658. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c3693>] :ip_conntrack:ip_conntrack_in+0x3c0/0x46d
  659. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875a23e>] :bridge:__br_forward+0x58/0x9c
  660. Jun 1 22:47:34 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  661. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  662. Jun 1 22:47:34 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  663. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  664. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875fd10>] :bridge:br_nf_pre_routing+0x5f0/0x61c
  665. Jun 1 22:47:34 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  666. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  667. Jun 1 22:47:34 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  668. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  669. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875b082>] :bridge:br_handle_frame+0x16e/0x1a4
  670. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026d76f>] do_IRQ+0xc8/0xd1
  671. Jun 1 22:47:34 hv004 kernel: [<ffffffff802214c5>] netif_receive_skb+0x3a8/0x4c4
  672. Jun 1 22:47:34 hv004 kernel: [<ffffffff88257d01>] :igb:igb_poll+0x72b/0xb55
  673. Jun 1 22:47:34 hv004 kernel: [<ffffffff8020d09d>] net_rx_action+0xb4/0x1c6
  674. Jun 1 22:47:34 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  675. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  676. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  677. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  678. Jun 1 22:47:34 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  679. Jun 1 22:47:34 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  680. Jun 1 22:47:34 hv004 kernel: [<ffffffff8029b4f9>] rcu_pending+0x26/0x50
  681. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  682. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  683. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  684. Jun 1 22:47:34 hv004 kernel:
  685. Jun 1 22:47:34 hv004 kernel: BUG: soft lockup - CPU#5 stuck for 60s! [events/5:79]
  686. Jun 1 22:47:34 hv004 kernel: CPU 5:
  687. Jun 1 22:47:34 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  688. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  689. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  690. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  691. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  692. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  693. Jun 1 22:47:34 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  694. Jun 1 22:47:34 hv004 kernel: Pid: 79, comm: events/5 Not tainted 2.6.18-274.7.1.el5xen #1
  695. Jun 1 22:47:34 hv004 kernel: RIP: e030:[<ffffffff802610a7>] [<ffffffff802610a7>] __write_lock_failed+0xf/0x20
  696. Jun 1 22:47:34 hv004 kernel: RSP: e02b:ffff88007ff63ee8 EFLAGS: 00000287
  697. Jun 1 22:47:34 hv004 kernel: RAX: ffff88007f817fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  698. Jun 1 22:47:34 hv004 kernel: RDX: ffffffffff578140 RSI: ffff880066786788 RDI: ffffffff888ce660
  699. Jun 1 22:47:34 hv004 kernel: RBP: ffff88007ff54000 R08: ffff880032e33c18 R09: 0000000000000005
  700. Jun 1 22:47:34 hv004 kernel: R10: ffff880066786788 R11: ffff8800667867a0 R12: ffff88001808cd90
  701. Jun 1 22:47:34 hv004 kernel: R13: 0000000000000100 R14: ffff88001808cd78 R15: ffffffff888c20fd
  702. Jun 1 22:47:34 hv004 kernel: FS: 00002b82dbd75f70(0000) GS:ffffffff80630280(0000) knlGS:0000000000000000
  703. Jun 1 22:47:34 hv004 kernel: CS: e033 DS: 0000 ES: 0000
  704. Jun 1 22:47:34 hv004 kernel:
  705. Jun 1 22:47:34 hv004 kernel: Call Trace:
  706. Jun 1 22:47:34 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  707. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  708. Jun 1 22:47:34 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  709. Jun 1 22:47:34 hv004 kernel: [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  710. Jun 1 22:47:34 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  711. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  712. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  713. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  714. Jun 1 22:47:34 hv004 kernel: <EOI> [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  715. Jun 1 22:47:34 hv004 kernel: [<ffffffff88252b88>] :igb:igb_update_stats+0x42a/0x6c3
  716. Jun 1 22:47:34 hv004 kernel: [<ffffffff882582dc>] :igb:igb_watchdog_task+0x1b1/0x331
  717. Jun 1 22:47:34 hv004 kernel: [<ffffffff8825812b>] :igb:igb_watchdog_task+0x0/0x331
  718. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024f152>] run_workqueue+0x9e/0xfb
  719. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024b9df>] worker_thread+0x0/0x122
  720. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024bacf>] worker_thread+0xf0/0x122
  721. Jun 1 22:47:34 hv004 kernel: [<ffffffff802895dd>] default_wake_function+0x0/0xe
  722. Jun 1 22:47:34 hv004 kernel: [<ffffffff8023398b>] kthread+0xfe/0x132
  723. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fb2c>] child_rip+0xa/0x12
  724. Jun 1 22:47:34 hv004 kernel: [<ffffffff8023388d>] kthread+0x0/0x132
  725. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fb22>] child_rip+0x0/0x12
  726. Jun 1 22:47:34 hv004 kernel:
  727. Jun 1 22:47:34 hv004 kernel: BUG: soft lockup - CPU#14 stuck for 60s! [swapper:0]
  728. Jun 1 22:47:34 hv004 kernel: CPU 14:
  729. Jun 1 22:47:34 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_ac
  730. l arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondem
  731. and acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp b
  732. nx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backli
  733. ght sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm ser
  734. io_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  735. Jun 1 22:47:34 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  736. Jun 1 22:47:34 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  737. Jun 1 22:47:34 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  738. Jun 1 22:47:34 hv004 kernel: RSP: e02b:ffff88007fe73b98 EFLAGS: 00000287
  739. Jun 1 22:47:34 hv004 kernel: RAX: ffff880007be9fd8 RBX: ffffffff888ce660 RCX: ffff8800392c5860
  740. Jun 1 22:47:34 hv004 kernel: RDX: ffff8800392c5803 RSI: 0000000000000002 RDI: ffffffff888ce660
  741. Jun 1 22:47:34 hv004 kernel: RBP: ffff88001b259528 R08: 00000000d00932f4 R09: 0000000000000000
  742. Jun 1 22:47:34 hv004 kernel: R10: ffff88001b259528 R11: 0000000000000130 R12: ffff88005d61d380
  743. Jun 1 22:47:34 hv004 kernel: R13: ffff88007fe73cd0 R14: ffffffff888ce820 R15: ffffffff808d7240
  744. Jun 1 22:47:34 hv004 kernel: FS: 00002adafe98eaf0(0000) GS:ffffffff80630700(0000) knlGS:0000000000000000
  745. Jun 1 22:47:34 hv004 kernel: CS: e033 DS: 002b ES: 002b
  746. Jun 1 22:47:34 hv004 kernel:
  747. Jun 1 22:47:34 hv004 kernel: Call Trace:
  748. Jun 1 22:47:34 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  749. Jun 1 22:47:34 hv004 kernel: [<ffffffff888c3437>] :ip_conntrack:ip_conntrack_in+0x164/0x46d
  750. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ae69>] :bridge:br_handle_frame_finish+0x128/0x1d3
  751. Jun 1 22:47:34 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  752. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  753. Jun 1 22:47:34 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  754. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  755. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875fd10>] :bridge:br_nf_pre_routing+0x5f0/0x61c
  756. Jun 1 22:47:34 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  757. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  758. Jun 1 22:47:34 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  759. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  760. Jun 1 22:47:34 hv004 kernel: [<ffffffff8875b082>] :bridge:br_handle_frame+0x16e/0x1a4
  761. Jun 1 22:47:34 hv004 kernel: [<ffffffff802214c5>] netif_receive_skb+0x3a8/0x4c4
  762. Jun 1 22:47:34 hv004 kernel: [<ffffffff88257d01>] :igb:igb_poll+0x72b/0xb55
  763. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026fc98>] timer_interrupt+0x3d2/0x3e6
  764. Jun 1 22:47:34 hv004 kernel: [<ffffffff803b72dd>] unmask_evtchn+0x2d/0xd9
  765. Jun 1 22:47:34 hv004 kernel: [<ffffffff8020d09d>] net_rx_action+0xb4/0x1c6
  766. Jun 1 22:47:34 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  767. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  768. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  769. Jun 1 22:47:34 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  770. Jun 1 22:47:34 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  771. Jun 1 22:47:34 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  772. Jun 1 22:47:34 hv004 kernel: [<ffffffff8029b4f9>] rcu_pending+0x26/0x50
  773. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  774. Jun 1 22:47:34 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  775. Jun 1 22:47:34 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  776. Jun 1 22:47:34 hv004 kernel:
  777. Jun 1 22:47:49 hv004 kernel: connection2:0: detected conn error (1011)
  778. Jun 1 22:47:49 hv004 multipathd: 8:64: reinstated
  779. Jun 1 22:47:49 hv004 multipathd: mpath6: remaining active paths: 1
  780. Jun 1 22:47:49 hv004 multipathd: sdl: readsector0 checker reports path is up
  781. Jun 1 22:47:49 hv004 multipathd: 8:176: reinstated
  782. Jun 1 22:47:49 hv004 multipathd: mpath7: remaining active paths: 1
  783. Jun 1 22:47:49 hv004 multipathd: sdp: readsector0 checker reports path is up
  784. Jun 1 22:47:49 hv004 multipathd: 8:240: reinstated
  785. Jun 1 22:47:49 hv004 multipathd: mpath8: remaining active paths: 1
  786. Jun 1 22:47:49 hv004 multipathd: sdq: readsector0 checker reports path is up
  787. Jun 1 22:47:49 hv004 multipathd: 65:0: reinstated
  788. Jun 1 22:47:49 hv004 multipathd: mpath9: remaining active paths: 1
  789. Jun 1 22:47:49 hv004 multipathd: sdr: readsector0 checker reports path is up
  790. Jun 1 22:47:49 hv004 multipathd: 65:16: reinstated
  791. Jun 1 22:47:49 hv004 multipathd: mpath10: remaining active paths: 1
  792. Jun 1 22:47:49 hv004 multipathd: sds: readsector0 checker reports path is up
  793. Jun 1 22:47:49 hv004 multipathd: 65:32: reinstated
  794. Jun 1 22:47:49 hv004 multipathd: mpath11: remaining active paths: 1
  795. Jun 1 22:47:49 hv004 multipathd: sdt: readsector0 checker reports path is up
  796. Jun 1 22:47:49 hv004 multipathd: 65:48: reinstated
  797. Jun 1 22:47:49 hv004 multipathd: mpath12: remaining active paths: 1
  798. Jun 1 22:47:49 hv004 multipathd: sdu: readsector0 checker reports path is up
  799. Jun 1 22:47:49 hv004 multipathd: 65:64: reinstated
  800. Jun 1 22:47:49 hv004 multipathd: mpath13: remaining active paths: 1
  801. Jun 1 22:47:49 hv004 multipathd: dm-2: add map (uevent)
  802. Jun 1 22:47:49 hv004 multipathd: dm-2: devmap already registered
  803. Jun 1 22:47:49 hv004 multipathd: dm-4: add map (uevent)
  804. Jun 1 22:47:49 hv004 multipathd: dm-4: devmap already registered
  805. Jun 1 22:47:49 hv004 multipathd: dm-6: add map (uevent)
  806. Jun 1 22:47:49 hv004 multipathd: dm-6: devmap already registered
  807. Jun 1 22:47:49 hv004 multipathd: dm-7: add map (uevent)
  808. Jun 1 22:47:49 hv004 multipathd: dm-7: devmap already registered
  809. Jun 1 22:47:49 hv004 multipathd: dm-8: add map (uevent)
  810. Jun 1 22:47:49 hv004 multipathd: dm-8: devmap already registered
  811. Jun 1 22:47:49 hv004 multipathd: dm-9: add map (uevent)
  812. Jun 1 22:47:49 hv004 multipathd: dm-9: devmap already registered
  813. Jun 1 22:47:49 hv004 multipathd: dm-10: add map (uevent)
  814. Jun 1 22:47:49 hv004 multipathd: dm-10: devmap already registered
  815. Jun 1 22:47:49 hv004 multipathd: dm-11: add map (uevent)
  816. Jun 1 22:47:49 hv004 multipathd: dm-11: devmap already registered
  817. Jun 1 22:47:50 hv004 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  818. Jun 1 22:47:53 hv004 iscsid: connection2:0 is operational after recovery (1 attempts)
  819. Jun 1 22:52:22 hv004 multipathd: dm-5: add map (uevent)
  820. Jun 1 22:52:22 hv004 multipathd: dm-5: devmap already registered
  821. Jun 1 22:52:22 hv004 kernel: device-mapper: table: 253:38: multipath: error getting device
  822. Jun 1 22:52:22 hv004 kernel: device-mapper: ioctl: error adding target to table
  823. Jun 1 22:52:22 hv004 multipathd: dm-38: remove map (uevent)
  824. Jun 1 22:52:22 hv004 multipathd: dm-38: devmap not registered, can't remove
  825. Jun 1 22:52:22 hv004 kernel: device-mapper: table: 253:38: multipath: error getting device
  826. Jun 1 22:52:22 hv004 kernel: device-mapper: ioctl: error adding target to table
  827. Jun 1 22:52:22 hv004 multipathd: dm-38: remove map (uevent)
  828. Jun 1 22:52:22 hv004 multipathd: dm-38: devmap not registered, can't remove
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!