API tools faq
paste
Advertisement
Guest User

hv004-01

a guest
Jun 12th, 2012
133
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 36.22 KB | None | 0 0
raw download clone embed print report
  1. Jun 3 01:40:19 hv004 kernel: connection1:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5762846770, last ping 5762848020, now 5762850520
  2. Jun 3 01:40:19 hv004 kernel: connection1:0: detected conn error (1011)
  3. Jun 3 01:40:20 hv004 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  4. Jun 3 01:40:22 hv004 multipathd: sdg: readsector0 checker reports path is down
  5. Jun 3 01:40:22 hv004 multipathd: checker failed path 8:96 in map mpath5
  6. Jun 3 01:40:22 hv004 kernel: device-mapper: multipath: Failing path 8:96.
  7. Jun 3 01:40:22 hv004 multipathd: mpath5: remaining active paths: 0
  8. Jun 3 01:40:22 hv004 multipathd: dm-5: add map (uevent)
  9. Jun 3 01:40:22 hv004 multipathd: dm-5: devmap already registered
  10. Jun 3 01:40:38 hv004 kernel: session1: session recovery timed out after 15 secs
  11. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  12. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 8875936280
  13. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  14. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 8872011296
  15. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  16. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 8872011256
  17. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  18. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 14380990536
  19. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  20. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 8871931104
  21. Jun 3 01:40:38 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  22. Jun 3 01:40:38 hv004 kernel: end_request: I/O error, dev sdg, sector 8882440336
  23. Jun 3 01:40:38 hv004 multipathd: sdb: readsector0 checker reports path is down
  24. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  25. Jun 3 01:40:40 hv004 multipathd: checker failed path 8:16 in map mpath4
  26. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8880387320
  27. Jun 3 01:40:40 hv004 multipathd: mpath4: remaining active paths: 0
  28. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  29. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8875931504
  30. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  31. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8875655928
  32. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  33. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8871930688
  34. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  35. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8762790016
  36. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  37. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8762593472
  38. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  39. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8762528208
  40. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  41. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8730546352
  42. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  43. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8730546312
  44. Jun 3 01:40:40 hv004 kernel: sd 5:0:0:1: SCSI error: return code = 0x000f0000
  45. Jun 3 01:40:40 hv004 kernel: end_request: I/O error, dev sdg, sector 8880326200
  46. Jun 3 01:40:40 hv004 kernel: device-mapper: multipath: Failing path 8:16.
  47. Jun 3 01:40:45 hv004 kernel: connection2:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5762851528, last ping 5762853611, now 5762856851
  48. Jun 3 01:40:45 hv004 kernel: connection2:0: detected conn error (1011)
  49. Jun 3 01:40:53 hv004 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  50. Jun 3 01:40:57 hv004 kernel: connection4:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5762855911, last ping 5762857330, now 5762859830
  51. Jun 3 01:40:57 hv004 kernel: connection4:0: detected conn error (1011)
  52. Jun 3 01:41:26 hv004 kernel: BUG: soft lockup - CPU#2 stuck for 60s! [swapper:0]
  53. Jun 3 01:41:26 hv004 kernel: CPU 2:
  54. Jun 3 01:41:26 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm serio_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  55. Jun 3 01:41:26 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  56. Jun 3 01:41:26 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  57. Jun 3 01:41:26 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  58. Jun 3 01:41:26 hv004 kernel: RSP: e02b:ffff88007ff13ee8 EFLAGS: 00000287
  59. Jun 3 01:41:26 hv004 kernel: RAX: ffff88007ffa9fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  60. Jun 3 01:41:26 hv004 kernel: RDX: ffffffffff578080 RSI: ffff88003a490d78 RDI: ffffffff888ce660
  61. Jun 3 01:41:26 hv004 kernel: RBP: ffff88007ff04000 R08: ffff880038b37818 R09: 0000000000000002
  62. Jun 3 01:41:26 hv004 kernel: R10: ffff88003a490d78 R11: ffff88003a490d90 R12: ffff88004b1182e0
  63. Jun 3 01:41:43 hv004 kernel: R13: 0000000000000100 R14: ffff88004b1182c8 R15: ffffffff888c20fd
  64. Jun 3 01:41:44 hv004 kernel: FS: 00002b0625827be0(0000) GS:ffffffff80630100(0000) knlGS:0000000000000000
  65. Jun 3 01:41:44 hv004 kernel: CS: e033 DS: 002b ES: 002b
  66. Jun 3 01:41:44 hv004 kernel:
  67. Jun 3 01:41:44 hv004 kernel: Call Trace:
  68. Jun 3 01:41:44 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  69. Jun 3 01:41:44 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  70. Jun 3 01:41:44 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  71. Jun 3 01:41:44 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  72. Jun 3 01:41:44 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  73. Jun 3 01:41:44 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  74. Jun 3 01:41:44 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  75. Jun 3 01:41:44 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  76. Jun 3 01:41:44 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  77. Jun 3 01:41:44 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  78. Jun 3 01:41:44 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  79. Jun 3 01:41:44 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  80. Jun 3 01:41:44 hv004 kernel:
  81. Jun 3 01:41:53 hv004 kernel: BUG: soft lockup - CPU#12 stuck for 60s! [swapper:0]
  82. Jun 3 01:41:53 hv004 kernel: CPU 12:
  83. Jun 3 01:41:53 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm serio_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  84. Jun 3 01:41:53 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  85. Jun 3 01:41:53 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  86. Jun 3 01:41:53 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  87. Jun 3 01:41:53 hv004 kernel: RSP: e02b:ffff88007fe378e8 EFLAGS: 00000287
  88. Jun 3 01:41:53 hv004 kernel: RAX: ffff880007be3fd8 RBX: ffffffff888ce660 RCX: 00000000f5d9fa6b
  89. Jun 3 01:41:53 hv004 kernel: RDX: 0000000000000276 RSI: 0000000000002000 RDI: ffffffff888ce660
  90. Jun 3 01:41:53 hv004 kernel: RBP: ffff88007332a788 R08: 00000000386739bb R09: ffffffff888cd960
  91. Jun 3 01:41:53 hv004 kernel: R10: 0000000000000002 R11: 0000000000000001 R12: 00000000000004de
  92. Jun 3 01:41:53 hv004 kernel: R13: ffff88007fe37a10 R14: ffff88007fe37a10 R15: ffffffff808d7280
  93. Jun 3 01:41:53 hv004 kernel: FS: 00002ae4f2aa96e0(0000) GS:ffffffff80630600(0000) knlGS:0000000000000000
  94. Jun 3 01:42:02 hv004 kernel: CS: e033 DS: 002b ES: 002b
  95. Jun 3 01:42:02 hv004 kernel:
  96. Jun 3 01:42:02 hv004 kernel: Call Trace:
  97. Jun 3 01:42:02 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  98. Jun 3 01:42:02 hv004 kernel: [<ffffffff888c2f0e>] :ip_conntrack:__ip_conntrack_confirm+0x56/0x2f8
  99. Jun 3 01:42:02 hv004 kernel: [<ffffffff888c115a>] :ip_conntrack:ip_confirm+0x29/0x39
  100. Jun 3 01:42:02 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  101. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875e8e3>] :bridge:br_nf_dev_queue_xmit+0x0/0x3b
  102. Jun 3 01:42:02 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  103. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875e8e3>] :bridge:br_nf_dev_queue_xmit+0x0/0x3b
  104. Jun 3 01:42:02 hv004 kernel: [<ffffffff88777bd0>] :ip_tables:ipt_do_table+0x2ca/0x2fa
  105. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ec1f>] :bridge:br_nf_post_routing+0x167/0x197
  106. Jun 3 01:42:02 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  107. Jun 3 01:42:02 hv004 kernel: [<ffffffff88759f95>] :bridge:br_dev_queue_push_xmit+0x0/0x200
  108. Jun 3 01:42:02 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  109. Jun 3 01:42:02 hv004 kernel: [<ffffffff88759f95>] :bridge:br_dev_queue_push_xmit+0x0/0x200
  110. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875a1d4>] :bridge:br_forward_finish+0x3f/0x51
  111. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875eab0>] :bridge:br_nf_forward_finish+0xf7/0xff
  112. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875f34e>] :bridge:br_nf_forward_ip+0x150/0x160
  113. Jun 3 01:42:02 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  114. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  115. Jun 3 01:42:02 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  116. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875a195>] :bridge:br_forward_finish+0x0/0x51
  117. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875a23e>] :bridge:__br_forward+0x58/0x9c
  118. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ae69>] :bridge:br_handle_frame_finish+0x128/0x1d3
  119. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875f123>] :bridge:br_nf_pre_routing_finish+0x2e9/0x2f8
  120. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  121. Jun 3 01:42:02 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  122. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ee3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8
  123. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875fd20>] :bridge:br_nf_pre_routing+0x600/0x61c
  124. Jun 3 01:42:02 hv004 kernel: [<ffffffff802352fe>] nf_iterate+0x41/0x7d
  125. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  126. Jun 3 01:42:02 hv004 kernel: [<ffffffff80258563>] nf_hook_slow+0x58/0xbc
  127. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875ad41>] :bridge:br_handle_frame_finish+0x0/0x1d3
  128. Jun 3 01:42:02 hv004 kernel: [<ffffffff8875b082>] :bridge:br_handle_frame+0x16e/0x1a4
  129. Jun 3 01:42:02 hv004 kernel: [<ffffffff802214c5>] netif_receive_skb+0x3a8/0x4c4
  130. Jun 3 01:42:02 hv004 kernel: [<ffffffff88257d01>] :igb:igb_poll+0x72b/0xb55
  131. Jun 3 01:42:03 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  132. Jun 3 01:42:03 hv004 kernel: [<ffffffff8020d09d>] net_rx_action+0xb4/0x1c6
  133. Jun 3 01:42:03 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  134. Jun 3 01:42:03 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  135. Jun 3 01:42:03 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  136. Jun 3 01:42:03 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  137. Jun 3 01:42:03 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  138. Jun 3 01:42:03 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  139. Jun 3 01:42:03 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  140. Jun 3 01:42:03 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  141. Jun 3 01:42:03 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  142. Jun 3 01:42:03 hv004 kernel:
  143. Jun 3 01:42:03 hv004 kernel: connection3:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5762870608, last ping 5762872170, now 5762875016
  144. Jun 3 01:42:03 hv004 kernel: connection3:0: detected conn error (1011)
  145. Jun 3 01:42:26 hv004 kernel: BUG: soft lockup - CPU#2 stuck for 60s! [swapper:0]
  146. Jun 3 01:42:26 hv004 kernel: CPU 2:
  147. Jun 3 01:42:26 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm serio_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  148. Jun 3 01:42:26 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  149. Jun 3 01:42:26 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  150. Jun 3 01:42:26 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  151. Jun 3 01:42:26 hv004 kernel: RSP: e02b:ffff88007ff13ee8 EFLAGS: 00000287
  152. Jun 3 01:42:26 hv004 kernel: RAX: ffff88007ffa9fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  153. Jun 3 01:42:26 hv004 kernel: RDX: ffffffffff578080 RSI: ffff880038b379c8 RDI: ffffffff888ce660
  154. Jun 3 01:42:26 hv004 kernel: RBP: ffff88007ff04000 R08: ffff880038b37818 R09: 0000000000000002
  155. Jun 3 01:42:26 hv004 kernel: R10: ffff880015230788 R11: ffff8800152307a0 R12: ffff88003e0ffa00
  156. Jun 3 01:42:26 hv004 kernel: R13: 0000000000000100 R14: ffff88003e0ff9e8 R15: ffffffff888c20fd
  157. Jun 3 01:42:26 hv004 kernel: FS: 00002b0625827be0(0000) GS:ffffffff80630100(0000) knlGS:0000000000000000
  158. Jun 3 01:42:26 hv004 kernel: CS: e033 DS: 002b ES: 002b
  159. Jun 3 01:42:26 hv004 kernel:
  160. Jun 3 01:42:26 hv004 kernel: Call Trace:
  161. Jun 3 01:42:26 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  162. Jun 3 01:42:26 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  163. Jun 3 01:42:26 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  164. Jun 3 01:42:26 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  165. Jun 3 01:42:26 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  166. Jun 3 01:42:26 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  167. Jun 3 01:42:26 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  168. Jun 3 01:42:26 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  169. Jun 3 01:42:26 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  170. Jun 3 01:42:26 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  171. Jun 3 01:42:26 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  172. Jun 3 01:42:26 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  173. Jun 3 01:42:26 hv004 kernel:
  174. Jun 3 01:42:55 hv004 kernel: BUG: soft lockup - CPU#11 stuck for 60s! [swapper:0]
  175. Jun 3 01:42:55 hv004 kernel: CPU 11:
  176. Jun 3 01:42:55 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm serio_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  177. Jun 3 01:42:55 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  178. Jun 3 01:42:55 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  179. Jun 3 01:42:55 hv004 kernel: RIP: e030:[<ffffffff802610a1>] [<ffffffff802610a1>] __write_lock_failed+0x9/0x20
  180. Jun 3 01:42:55 hv004 kernel: RSP: e02b:ffff88007fe23ee8 EFLAGS: 00000287
  181. Jun 3 01:42:55 hv004 kernel: RAX: ffff880007be1fd8 RBX: ffffffff888ce660 RCX: ffff880001574638
  182. Jun 3 01:42:55 hv004 kernel: RDX: ffffffffff5782c0 RSI: ffff88007fe23ee0 RDI: ffffffff888ce660
  183. Jun 3 01:42:55 hv004 kernel: RBP: ffff880001574000 R08: ffff8800222efec0 R09: 0000000000000000
  184. Jun 3 01:42:55 hv004 kernel: R10: ffff88003f042480 R11: ffff8800222efec0 R12: ffff8800222efec0
  185. Jun 3 01:42:55 hv004 kernel: R13: 0000000000000100 R14: ffff8800222efea8 R15: ffffffff888c20fd
  186. Jun 3 01:42:55 hv004 kernel: FS: 00002b4701d08170(0000) GS:ffffffff80630580(0000) knlGS:0000000000000000
  187. Jun 3 01:42:55 hv004 kernel: CS: e033 DS: 002b ES: 002b
  188. Jun 3 01:42:55 hv004 kernel:
  189. Jun 3 01:42:55 hv004 kernel: Call Trace:
  190. Jun 3 01:42:55 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  191. Jun 3 01:42:55 hv004 kernel: [<ffffffff888c210d>] :ip_conntrack:death_by_timeout+0x10/0x6c
  192. Jun 3 01:42:55 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  193. Jun 3 01:42:55 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  194. Jun 3 01:42:55 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  195. Jun 3 01:42:55 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  196. Jun 3 01:42:55 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  197. Jun 3 01:42:55 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  198. Jun 3 01:42:55 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  199. Jun 3 01:42:55 hv004 kernel: [<ffffffff8029b4f9>] rcu_pending+0x26/0x50
  200. Jun 3 01:42:55 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  201. Jun 3 01:42:55 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  202. Jun 3 01:42:55 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  203. Jun 3 01:42:55 hv004 kernel:
  204. Jun 3 01:43:14 hv004 iscsid: Kernel reported iSCSI connection 4:0 error (1011) state (3)
  205. Jun 3 01:43:26 hv004 kernel: BUG: soft lockup - CPU#2 stuck for 60s! [swapper:0]
  206. Jun 3 01:43:26 hv004 kernel: CPU 2:
  207. Jun 3 01:43:26 hv004 kernel: Modules linked in: fuse tun ipt_iprange xt_multiport xt_state ip_conntrack nfnetlink xt_physdev sch_tbf ext4 jbd2 crc16 nls_utf8 hfsplus nfs nfs_acl arptable_filter arp_tables ip6table_mangle ipt_REJECT iptable_filter ip_tables bridge netloop netbk blktap blkbk autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac parport_pc lp parport loop joydev sg pcspkr i2c_i801 i2c_core igb 8021q dca tpm_tis serial_core tpm serio_raw tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata shp
  208. Jun 3 01:43:26 hv004 kernel: hp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  209. Jun 3 01:43:26 hv004 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-274.7.1.el5xen #1
  210. Jun 3 01:43:26 hv004 kernel: RIP: e030:[<ffffffff802610a7>] [<ffffffff802610a7>] __write_lock_failed+0xf/0x20
  211. Jun 3 01:43:26 hv004 kernel: RSP: e02b:ffff88007ff13ee8 EFLAGS: 00000287
  212. Jun 3 01:43:26 hv004 kernel: RAX: ffff88007ffa9fd8 RBX: ffffffff888ce660 RCX: 0000000000000000
  213. Jun 3 01:43:26 hv004 kernel: RDX: ffff88005a8768b8 RSI: 0000000000000004 RDI: ffffffff888ce660
  214. Jun 3 01:43:26 hv004 kernel: RBP: ffff88007ff04000 R08: ffff880038b37818 R09: 0000000000000002
  215. Jun 3 01:43:26 hv004 kernel: R10: ffff88003ea1d198 R11: ffffffff888c31e4 R12: ffff88005a8768d0
  216. Jun 3 01:43:26 hv004 kernel: R13: 0000000000000100 R14: ffff88005a8768b8 R15: ffffffff888c20fd
  217. Jun 3 01:43:26 hv004 kernel: FS: 00002b0625827be0(0000) GS:ffffffff80630100(0000) knlGS:0000000000000000
  218. Jun 3 01:43:27 hv004 kernel: CS: e033 DS: 002b ES: 002b
  219. Jun 3 01:43:27 hv004 kernel:
  220. Jun 3 01:43:27 hv004 kernel: Call Trace:
  221. Jun 3 01:43:27 hv004 kernel: <IRQ> [<ffffffff80263a11>] _write_lock_bh+0x1a/0x1c
  222. Jun 3 01:43:27 hv004 kernel: [<ffffffff888c3250>] :ip_conntrack:destroy_conntrack+0x6c/0xde
  223. Jun 3 01:43:27 hv004 kernel: [<ffffffff8029415d>] run_timer_softirq+0x191/0x242
  224. Jun 3 01:43:27 hv004 kernel: [<ffffffff80212f28>] __do_softirq+0x8d/0x13b
  225. Jun 3 01:43:27 hv004 kernel: [<ffffffff8025fda4>] call_softirq+0x1c/0x278
  226. Jun 3 01:43:27 hv004 kernel: [<ffffffff8026db69>] do_softirq+0x31/0x90
  227. Jun 3 01:43:27 hv004 kernel: [<ffffffff8025f8d6>] do_hypervisor_callback+0x1e/0x2c
  228. Jun 3 01:43:27 hv004 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  229. Jun 3 01:43:27 hv004 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  230. Jun 3 01:43:27 hv004 kernel: [<ffffffff8026efa8>] raw_safe_halt+0x87/0xab
  231. Jun 3 01:43:27 hv004 kernel: [<ffffffff8026c553>] xen_idle+0x38/0x4a
  232. Jun 3 01:43:28 hv004 kernel: [<ffffffff8024ac6d>] cpu_idle+0x97/0xba
  233. Jun 3 01:43:28 hv004 kernel:
  234. Jun 3 01:43:28 hv004 kernel: session2: session recovery timed out after 15 secs
  235. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  236. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8875696800
  237. Jun 3 01:43:28 hv004 kernel: device-mapper: multipath: Failing path 8:112.
  238. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  239. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8882440344
  240. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  241. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8872011480
  242. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  243. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8730546352
  244. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  245. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8880326192
  246. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  247. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8762790288
  248. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  249. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 14380990536
  250. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  251. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8883025000
  252. Jun 3 01:43:28 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  253. Jun 3 01:43:28 hv004 kernel: end_request: I/O error, dev sdh, sector 8872011264
  254. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  255. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8875936280
  256. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  257. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8880326200
  258. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  259. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8875931512
  260. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  261. Jun 3 01:43:29 hv004 multipathd: sde: readsector0 checker reports path is down
  262. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8762528288
  263. Jun 3 01:43:29 hv004 multipathd: checker failed path 8:64 in map mpath6
  264. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  265. Jun 3 01:43:29 hv004 multipathd: mpath6: remaining active paths: 0
  266. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8871931104
  267. Jun 3 01:43:29 hv004 multipathd: sdg: readsector0 checker reports path is down
  268. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  269. Jun 3 01:43:29 hv004 multipathd: dm-3: add map (uevent)
  270. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8762790016
  271. Jun 3 01:43:29 hv004 multipathd: dm-3: devmap already registered
  272. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  273. Jun 3 01:43:29 hv004 multipathd: dm-5: add map (uevent)
  274. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8875655928
  275. Jun 3 01:43:29 hv004 multipathd: dm-5: devmap already registered
  276. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  277. Jun 3 01:43:29 hv004 multipathd: dm-2: add map (uevent)
  278. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8875931504
  279. Jun 3 01:43:29 hv004 multipathd: dm-2: devmap already registered
  280. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  281. Jun 3 01:43:29 hv004 multipathd: 8:112: mark as failed
  282. Jun 3 01:43:29 hv004 kernel: end_request: I/O error, dev sdh, sector 8875927136
  283. Jun 3 01:43:29 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  284. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8871930688
  285. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  286. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762528208
  287. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  288. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762593472
  289. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  290. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8872011256
  291. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  292. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8872011296
  293. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  294. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762790352
  295. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  296. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762528192
  297. Jun 3 01:43:30 hv004 iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3)
  298. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  299. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8880387320
  300. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  301. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8872011472
  302. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  303. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762790024
  304. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  305. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8762602872
  306. Jun 3 01:43:30 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  307. Jun 3 01:43:30 hv004 kernel: end_request: I/O error, dev sdh, sector 8882440336
  308. Jun 3 01:43:31 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  309. Jun 3 01:43:31 hv004 kernel: end_request: I/O error, dev sdh, sector 8730546336
  310. Jun 3 01:43:31 hv004 kernel: sd 6:0:0:1: SCSI error: return code = 0x000f0000
  311. Jun 3 01:43:31 hv004 kernel: end_request: I/O error, dev sdh, sector 8730546312
  312. Jun 3 01:43:31 hv004 kernel: sd 8:0:0:0: timing out command, waited 60s
  313. Jun 3 01:43:31 hv004 kernel: device-mapper: multipath: Failing path 8:64.
  314. Jun 3 01:43:34 hv004 multipathd: sdb: readsector0 checker reports path is down
  315. Jun 3 01:43:34 hv004 multipathd: sdg: readsector0 checker reports path is down
  316. Jun 3 01:43:34 hv004 kernel: connection1:0: detected conn error (1011)
  317. Jun 3 01:43:35 hv004 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (2)
  318. Jun 3 01:43:37 hv004 iscsid: connection4:0 is operational after recovery (1 attempts)
  319. Jun 3 01:43:44 hv004 kernel: session3: session recovery timed out after 15 secs
  320. Jun 3 01:43:44 hv004 kernel: sd 7:0:0:1: SCSI error: return code = 0x000f0000
  321. Jun 3 01:43:44 hv004 kernel: end_request: I/O error, dev sdf, sector 3896813528
  322. Jun 3 01:43:44 hv004 kernel: sd 7:0:0:4: SCSI error: return code = 0x000f0000
  323. Jun 3 01:43:44 hv004 kernel: end_request: I/O error, dev sdk, sector 2957666936
  324. Jun 3 01:43:44 hv004 kernel: device-mapper: multipath: Failing path 8:160.
  325. Jun 3 01:43:44 hv004 kernel: device-mapper: multipath: Failing path 8:80.
  326. Jun 3 01:43:50 hv004 iscsid: connection3:0 is operational after recovery (1 attempts)
  327. Jun 3 01:43:52 hv004 iscsid: connection2:0 is operational after recovery (1 attempts)
  328. Jun 3 01:44:00 hv004 multipathd: sde: readsector0 checker reports path is up
  329. Jun 3 01:44:00 hv004 multipathd: 8:64: reinstated
  330. Jun 3 01:44:00 hv004 multipathd: mpath6: remaining active paths: 1
  331. Jun 3 01:44:02 hv004 multipathd: dm-8: add map (uevent)
  332. Jun 3 01:44:02 hv004 multipathd: dm-8: devmap already registered
  333. Jun 3 01:44:02 hv004 multipathd: dm-4: add map (uevent)
  334. Jun 3 01:44:02 hv004 multipathd: dm-4: devmap already registered
  335. Jun 3 01:44:02 hv004 multipathd: dm-2: add map (uevent)
  336. Jun 3 01:44:02 hv004 multipathd: dm-2: devmap already registered
  337. Jun 3 01:44:02 hv004 multipathd: 8:80: mark as failed
  338. Jun 3 01:44:02 hv004 multipathd: 8:160: mark as failed
  339. Jun 3 01:44:03 hv004 iscsid: connection1:0 is operational after recovery (2 attempts)
  340. Jun 3 01:44:05 hv004 multipathd: sdb: readsector0 checker reports path is up
  341. Jun 3 01:44:05 hv004 multipathd: 8:16: reinstated
  342. Jun 3 01:44:05 hv004 multipathd: mpath4: remaining active paths: 1
  343. Jun 3 01:44:05 hv004 multipathd: sdg: readsector0 checker reports path is up
  344. Jun 3 01:44:05 hv004 multipathd: 8:96: reinstated
  345. Jun 3 01:44:05 hv004 multipathd: mpath5: remaining active paths: 1
  346. Jun 3 01:44:05 hv004 multipathd: dm-3: add map (uevent)
  347. Jun 3 01:44:05 hv004 multipathd: dm-3: devmap already registered
  348. Jun 3 01:44:05 hv004 multipathd: dm-5: add map (uevent)
  349. Jun 3 01:44:05 hv004 multipathd: dm-5: devmap already registered
  350. Jun 3 01:44:05 hv004 multipathd: 8:112: mark as failed
  351. Jun 3 01:51:14 hv004 kernel: connection3:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5763000641, last ping 5763001937, now 5763014099
  352. Jun 3 01:51:14 hv004 kernel: connection3:0: detected conn error (1011)
  353. Jun 3 01:51:14 hv004 iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3)
  354. Jun 3 01:51:17 hv004 iscsid: connection3:0 is operational after recovery (1 attempts)
  355. Jun 3 01:55:21 hv004 kernel: connection3:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5763072196, last ping 5763073446, now 5763076010
  356. Jun 3 01:55:21 hv004 kernel: connection3:0: detected conn error (1011)
  357. Jun 3 01:55:22 hv004 kernel: NETDEV WATCHDOG: eth1: transmit timed out
  358. Jun 3 01:55:22 hv004 kernel: igb 0000:01:00.1: eth1: Reset adapter
  359. Jun 3 01:55:22 hv004 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
  360. Jun 3 01:55:24 hv004 iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3)
  361. Jun 3 01:55:29 hv004 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
  362. Jun 3 01:55:29 hv004 kernel: bonding: bond1: link status definitely up for interface eth1.
  363. Jun 3 01:55:40 hv004 kernel: session3: session recovery timed out after 15 secs
  364. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:3: SCSI error: return code = 0x000f0000
  365. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdj, sector 81093656
  366. Jun 3 01:55:40 hv004 kernel: device-mapper: multipath: Failing path 8:144.
  367. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:3: SCSI error: return code = 0x000f0000
  368. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdj, sector 83694336
  369. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:3: SCSI error: return code = 0x000f0000
  370. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdj, sector 69271544
  371. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:5: SCSI error: return code = 0x000f0000
  372. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdm, sector 1725247015
  373. Jun 3 01:55:40 hv004 kernel: device-mapper: multipath: Failing path 8:192.
  374. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:5: SCSI error: return code = 0x000f0000
  375. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdm, sector 4155769975
  376. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:5: SCSI error: return code = 0x000f0000
  377. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdm, sector 3948992743
  378. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:5: SCSI error: return code = 0x000f0000
  379. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdm, sector 4019821487
  380. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:6: SCSI error: return code = 0x000f0000
  381. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdn, sector 216372464
  382. Jun 3 01:55:40 hv004 kernel: device-mapper: multipath: Failing path 8:208.
  383. Jun 3 01:55:40 hv004 kernel: sd 7:0:0:6: SCSI error: return code = 0x000f0000
  384. Jun 3 01:55:40 hv004 kernel: end_request: I/O error, dev sdn, sector 1665354792
  385. Jun 3 01:55:45 hv004 kernel: connection1:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5763078146, last ping 5763079415, now 5763081921
  386. Jun 3 01:55:45 hv004 kernel: connection1:0: detected conn error (1011)
  387. Jun 3 01:55:49 hv004 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
  388. Jun 3 01:55:52 hv004 kernel: connection2:0: ping timeout of 10 secs expired, recv timeout 5, last rx 5763079553, last ping 5763080803, now 5763083752
  389. Jun 3 01:55:52 hv004 kernel: connection2:0: detected conn error (1011)
  390. Jun 3 01:55:53 hv004 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
  391. Jun 3 01:55:54 hv004 iscsid: connection3:0 is operational after recovery (1 attempts)
  392. Jun 3 01:55:55 hv004 multipathd: dm-7: add map (uevent)
  393. Jun 3 01:55:55 hv004 multipathd: dm-7: devmap already registered
  394. Jun 3 01:55:55 hv004 multipathd: 8:144: mark as failed
  395. Jun 3 01:55:55 hv004 multipathd: 8:208: mark as failed
  396. Jun 3 01:55:55 hv004 multipathd: dm-9: add map (uevent)
  397. Jun 3 01:55:55 hv004 multipathd: dm-9: devmap already registered
  398. Jun 3 01:55:55 hv004 multipathd: dm-10: add map (uevent)
  399. Jun 3 01:55:55 hv004 multipathd: dm-10: devmap already registered
  400. Jun 3 01:55:55 hv004 multipathd: 8:192: mark as failed
  401. Jun 3 01:55:57 hv004 iscsid: connection2:0 is operational after recovery (1 attempts)
  402. Jun 3 01:55:57 hv004 iscsid: connection1:0 is operational after recovery (1 attempts)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!