API tools faq
paste
Advertisement
treydock

Untitled

treydock
Jan 27th, 2014
381
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 37.44 KB | None | 0 0
raw download clone embed print report
  1. Jan 25 06:11:56 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  2. Jan 25 06:12:06 vm01 sanlock[2462]: 2014-01-25 06:12:06-0600 48071 [10244]: s3 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  3. Jan 25 06:12:06 vm01 sanlock[2462]: 2014-01-25 06:12:06-0600 48071 [10244]: s3 renewal error -202 delta_length 10 last_success 48041
  4. Jan 25 06:12:17 vm01 sanlock[2462]: 2014-01-25 06:12:17-0600 48082 [10244]: s3 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  5. Jan 25 06:12:17 vm01 sanlock[2462]: 2014-01-25 06:12:17-0600 48082 [10244]: s3 renewal error -202 delta_length 10 last_success 48041
  6. Jan 25 06:12:26 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  7. Jan 25 06:12:28 vm01 sanlock[2462]: 2014-01-25 06:12:28-0600 48093 [10244]: s3 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  8. Jan 25 06:12:28 vm01 sanlock[2462]: 2014-01-25 06:12:28-0600 48093 [10244]: s3 renewal error -202 delta_length 10 last_success 48041
  9. Jan 25 06:12:29 vm01 kernel: ------------[ cut here ]------------
  10. Jan 25 06:12:29 vm01 kernel: WARNING: at kernel/softirq.c:159 local_bh_enable_ip+0x7d/0xb0() (Not tainted)
  11. Jan 25 06:12:29 vm01 kernel: Hardware name: H8DMT-IBX
  12. Jan 25 06:12:29 vm01 kernel: Modules linked in: ebt_arp xprtrdma nfs fscache auth_rpcgss nfs_acl bonding 8021q garp ebtable_nat ebtables softdog lockd sunrpc powernow_k8 freq_table mperf bridge stp llc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 dm_round_robin dm_multipath vhost_net macvtap macvlan tun kvm_amd kvm microcode serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd sg igb dca i2c_algo_bit ptp pps_core mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core i2c_nforce2 i2c_core ext4 jbd2 mbcache raid1 sd_mod crc_t10dif pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
  13. Jan 25 06:12:29 vm01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.3.1.el6.x86_64 #1
  14. Jan 25 06:12:29 vm01 kernel: Call Trace:
  15. Jan 25 06:12:29 vm01 kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
  16. Jan 25 06:12:29 vm01 kernel: [<ffffffff81071e7a>] ? warn_slowpath_null+0x1a/0x20
  17. Jan 25 06:12:29 vm01 kernel: [<ffffffff8107a3ed>] ? local_bh_enable_ip+0x7d/0xb0
  18. Jan 25 06:12:29 vm01 kernel: [<ffffffff8152a4fb>] ? _spin_unlock_bh+0x1b/0x20
  19. Jan 25 06:12:29 vm01 kernel: [<ffffffffa043e4f0>] ? rpc_wake_up_status+0x70/0x80 [sunrpc]
  20. Jan 25 06:12:29 vm01 kernel: [<ffffffffa043779c>] ? xprt_wake_pending_tasks+0x2c/0x30 [sunrpc]
  21. Jan 25 06:12:29 vm01 kernel: [<ffffffffa04152fc>] ? rpcrdma_conn_func+0x9c/0xb0 [xprtrdma]
  22. Jan 25 06:12:29 vm01 kernel: [<ffffffffa0418450>] ? rpcrdma_qp_async_error_upcall+0x40/0x80 [xprtrdma]
  23. Jan 25 06:12:29 vm01 kernel: [<ffffffffa01861cb>] ? mlx4_ib_qp_event+0x8b/0x100 [mlx4_ib]
  24. Jan 25 06:12:29 vm01 kernel: [<ffffffffa012dc54>] ? mlx4_qp_event+0x74/0xf0 [mlx4_core]
  25. Jan 25 06:12:29 vm01 kernel: [<ffffffff81065c5e>] ? try_to_wake_up+0x24e/0x3e0
  26. Jan 25 06:12:29 vm01 kernel: [<ffffffffa011b057>] ? mlx4_eq_int+0x557/0xcb0 [mlx4_core]
  27. Jan 25 06:12:29 vm01 kernel: [<ffffffff811d5e10>] ? timeout_func+0x0/0x20
  28. Jan 25 06:12:29 vm01 kernel: [<ffffffff81065e45>] ? wake_up_process+0x15/0x20
  29. Jan 25 06:12:29 vm01 kernel: [<ffffffff811d5e29>] ? timeout_func+0x19/0x20
  30. Jan 25 06:12:29 vm01 kernel: [<ffffffffa011b7c4>] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core]
  31. Jan 25 06:12:29 vm01 kernel: [<ffffffff810e6ed0>] ? handle_IRQ_event+0x60/0x170
  32. Jan 25 06:12:29 vm01 kernel: [<ffffffff810e982e>] ? handle_edge_irq+0xde/0x180
  33. Jan 25 06:12:29 vm01 kernel: [<ffffffff8100faf9>] ? handle_irq+0x49/0xa0
  34. Jan 25 06:12:29 vm01 kernel: [<ffffffff81530fec>] ? do_IRQ+0x6c/0xf0
  35. Jan 25 06:12:29 vm01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  36. Jan 25 06:12:29 vm01 kernel: <EOI> [<ffffffff8103eacb>] ? native_safe_halt+0xb/0x10
  37. Jan 25 06:12:29 vm01 kernel: [<ffffffff810167bd>] ? default_idle+0x4d/0xb0
  38. Jan 25 06:12:29 vm01 kernel: [<ffffffff810168bd>] ? c1e_idle+0x9d/0x120
  39. Jan 25 06:12:29 vm01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  40. Jan 25 06:12:29 vm01 kernel: [<ffffffff8150cc1a>] ? rest_init+0x7a/0x80
  41. Jan 25 06:12:29 vm01 kernel: [<ffffffff81c26f8f>] ? start_kernel+0x424/0x430
  42. Jan 25 06:12:29 vm01 kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129
  43. Jan 25 06:12:29 vm01 kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124
  44. Jan 25 06:12:29 vm01 kernel: ---[ end trace 9cbf577d8c859b04 ]---
  45. Jan 25 06:12:29 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  46. Jan 25 06:12:36 vm01 sanlock[2462]: 2014-01-25 06:12:36-0600 48101 [2462]: s3 check_our_lease warning 60 last_success 48041
  47. Jan 25 06:12:37 vm01 sanlock[2462]: 2014-01-25 06:12:37-0600 48102 [2462]: s3 check_our_lease warning 61 last_success 48041
  48. Jan 25 06:12:38 vm01 sanlock[2462]: 2014-01-25 06:12:38-0600 48103 [2462]: s3 check_our_lease warning 62 last_success 48041
  49. Jan 25 06:12:39 vm01 sanlock[2462]: 2014-01-25 06:12:39-0600 48104 [10244]: s3 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  50. Jan 25 06:12:39 vm01 sanlock[2462]: 2014-01-25 06:12:39-0600 48104 [10244]: s3 renewal error -202 delta_length 10 last_success 48041
  51. Jan 25 06:12:39 vm01 sanlock[2462]: 2014-01-25 06:12:39-0600 48104 [2462]: s3 check_our_lease warning 63 last_success 48041
  52. Jan 25 06:12:40 vm01 sanlock[2462]: 2014-01-25 06:12:40-0600 48105 [2462]: s3 check_our_lease warning 64 last_success 48041
  53. Jan 25 06:12:41 vm01 sanlock[2462]: 2014-01-25 06:12:41-0600 48106 [2462]: s3 check_our_lease warning 65 last_success 48041
  54. Jan 25 06:12:42 vm01 sanlock[2462]: 2014-01-25 06:12:42-0600 48107 [2462]: s3 check_our_lease warning 66 last_success 48041
  55. Jan 25 06:12:43 vm01 sanlock[2462]: 2014-01-25 06:12:43-0600 48108 [2462]: s3 check_our_lease warning 67 last_success 48041
  56. Jan 25 06:12:44 vm01 sanlock[2462]: 2014-01-25 06:12:44-0600 48109 [2462]: s3 check_our_lease warning 68 last_success 48041
  57. Jan 25 06:12:45 vm01 sanlock[2462]: 2014-01-25 06:12:45-0600 48110 [2462]: s3 check_our_lease warning 69 last_success 48041
  58. Jan 25 06:12:46 vm01 sanlock[2462]: 2014-01-25 06:12:46-0600 48111 [2462]: s3 check_our_lease warning 70 last_success 48041
  59. Jan 25 06:12:46 vm01 wdmd[2451]: test warning now 48112 ping 48102 close 0 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  60. Jan 25 06:12:46 vm01 wdmd[2451]: /dev/watchdog closed unclean
  61. Jan 25 06:12:46 vm01 kernel: SoftDog: Unexpected close, not stopping watchdog!
  62. Jan 25 06:12:47 vm01 sanlock[2462]: 2014-01-25 06:12:47-0600 48112 [2462]: s3 check_our_lease warning 71 last_success 48041
  63. Jan 25 06:12:47 vm01 wdmd[2451]: test warning now 48113 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  64. Jan 25 06:12:48 vm01 sanlock[2462]: 2014-01-25 06:12:48-0600 48113 [2462]: s3 check_our_lease warning 72 last_success 48041
  65. Jan 25 06:12:48 vm01 wdmd[2451]: test warning now 48114 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  66. Jan 25 06:12:49 vm01 sanlock[2462]: 2014-01-25 06:12:49-0600 48114 [2462]: s3 check_our_lease warning 73 last_success 48041
  67. Jan 25 06:12:49 vm01 wdmd[2451]: test warning now 48115 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  68. Jan 25 06:12:50 vm01 sanlock[2462]: 2014-01-25 06:12:50-0600 48115 [10244]: s3 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  69. Jan 25 06:12:50 vm01 sanlock[2462]: 2014-01-25 06:12:50-0600 48115 [10244]: s3 renewal error -2 delta_length 10 last_success 48041
  70. Jan 25 06:12:50 vm01 sanlock[2462]: 2014-01-25 06:12:50-0600 48115 [2462]: s3 check_our_lease warning 74 last_success 48041
  71. Jan 25 06:12:50 vm01 wdmd[2451]: test warning now 48116 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  72. Jan 25 06:12:51 vm01 sanlock[2462]: 2014-01-25 06:12:51-0600 48116 [2462]: s3 check_our_lease warning 75 last_success 48041
  73. Jan 25 06:12:51 vm01 wdmd[2451]: test warning now 48117 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  74. Jan 25 06:12:52 vm01 sanlock[2462]: 2014-01-25 06:12:52-0600 48117 [2462]: s3 check_our_lease warning 76 last_success 48041
  75. Jan 25 06:12:52 vm01 wdmd[2451]: test warning now 48118 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  76. Jan 25 06:12:53 vm01 sanlock[2462]: 2014-01-25 06:12:53-0600 48118 [2462]: s3 check_our_lease warning 77 last_success 48041
  77. Jan 25 06:12:53 vm01 wdmd[2451]: test warning now 48119 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  78. Jan 25 06:12:54 vm01 sanlock[2462]: 2014-01-25 06:12:54-0600 48119 [2462]: s3 check_our_lease warning 78 last_success 48041
  79. Jan 25 06:12:54 vm01 wdmd[2451]: test warning now 48120 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  80. Jan 25 06:12:55 vm01 sanlock[2462]: 2014-01-25 06:12:55-0600 48120 [2462]: s3 check_our_lease warning 79 last_success 48041
  81. Jan 25 06:12:55 vm01 wdmd[2451]: test failed rem 51 now 48121 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  82. Jan 25 06:12:56 vm01 sanlock[2462]: 2014-01-25 06:12:56-0600 48121 [2462]: s3 check_our_lease failed 80
  83. Jan 25 06:12:56 vm01 sanlock[2462]: 2014-01-25 06:12:56-0600 48121 [2462]: s3 kill 2953 sig 15 count 1
  84. Jan 25 06:12:56 vm01 wdmd[2451]: test failed rem 50 now 48122 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  85. Jan 25 06:12:56 vm01 sanlock[2462]: 2014-01-25 06:12:56-0600 48122 [2462]: s3 kill 2953 sig 15 count 2
  86. Jan 25 06:12:57 vm01 wdmd[2451]: test failed rem 49 now 48123 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  87. Jan 25 06:12:57 vm01 sanlock[2462]: 2014-01-25 06:12:57-0600 48123 [2462]: s3 kill 2953 sig 15 count 3
  88. Jan 25 06:12:58 vm01 wdmd[2451]: test failed rem 48 now 48124 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  89. Jan 25 06:12:58 vm01 sanlock[2462]: 2014-01-25 06:12:58-0600 48124 [2462]: s3 kill 2953 sig 15 count 4
  90. Jan 25 06:12:59 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  91. Jan 25 06:12:59 vm01 wdmd[2451]: test failed rem 47 now 48125 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  92. Jan 25 06:12:59 vm01 sanlock[2462]: 2014-01-25 06:12:59-0600 48125 [2462]: s3 kill 2953 sig 15 count 5
  93. Jan 25 06:13:00 vm01 wdmd[2451]: test failed rem 46 now 48126 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  94. Jan 25 06:13:00 vm01 sanlock[2462]: 2014-01-25 06:13:00-0600 48126 [10244]: s3 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  95. Jan 25 06:13:00 vm01 sanlock[2462]: 2014-01-25 06:13:00-0600 48126 [10244]: s3 renewal error -2 delta_length 10 last_success 48041
  96. Jan 25 06:13:00 vm01 sanlock[2462]: 2014-01-25 06:13:00-0600 48126 [2462]: s3 kill 2953 sig 15 count 6
  97. Jan 25 06:13:01 vm01 wdmd[2451]: test failed rem 45 now 48127 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  98. Jan 25 06:13:01 vm01 sanlock[2462]: 2014-01-25 06:13:01-0600 48127 [2462]: s3 kill 2953 sig 15 count 7
  99. Jan 25 06:13:02 vm01 wdmd[2451]: test failed rem 44 now 48128 ping 48102 close 48112 renewal 48041 expire 48121 client 2462 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:1
  100. Jan 25 06:13:02 vm01 sanlock[2462]: 2014-01-25 06:13:02-0600 48128 [2462]: s3 kill 2953 sig 15 count 8
  101. Jan 25 06:13:02 vm01 sanlock[2462]: 2014-01-25 06:13:02-0600 48128 [2462]: dead 2953 ci 3 count 8
  102. Jan 25 06:13:02 vm01 respawn: slave '/usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid' died, respawning slave
  103. Jan 25 06:13:03 vm01 sanlock[2462]: 2014-01-25 06:13:03-0600 48128 [2462]: s3 all pids clear
  104. Jan 25 06:13:04 vm01 wdmd[2451]: /dev/watchdog reopen
  105. Jan 25 06:13:04 vm01 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012 File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012 s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
  106. Jan 25 06:13:04 vm01 vdsm vds WARNING Unable to load the json rpc server module. Please make sure it is installed.
  107. Jan 25 06:13:04 vm01 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
  108. Jan 25 06:13:04 vm01 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
  109. Jan 25 06:13:05 vm01 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::_readPauseCode unsupported by libvirt vm
  110. Jan 25 06:13:05 vm01 kernel: ata2: hard resetting link
  111. Jan 25 06:13:05 vm01 kernel: ata2: nv: skipping hardreset on occupied port
  112. Jan 25 06:13:05 vm01 kernel: ata3: hard resetting link
  113. Jan 25 06:13:05 vm01 kernel: ata4: hard resetting link
  114. Jan 25 06:13:05 vm01 kernel: ata5: soft resetting link
  115. Jan 25 06:13:05 vm01 kernel: ata6: EH complete
  116. Jan 25 06:13:05 vm01 kernel: ata1: hard resetting link
  117. Jan 25 06:13:05 vm01 kernel: ata1: nv: skipping hardreset on occupied port
  118. Jan 25 06:13:05 vm01 kernel: ata5: EH complete
  119. Jan 25 06:13:05 vm01 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  120. Jan 25 06:13:05 vm01 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  121. Jan 25 06:13:05 vm01 kernel: ata2.00: configured for UDMA/133
  122. Jan 25 06:13:05 vm01 kernel: ata2: EH complete
  123. Jan 25 06:13:05 vm01 kernel: ata1.00: configured for UDMA/133
  124. Jan 25 06:13:05 vm01 kernel: ata1: EH complete
  125. Jan 25 06:13:06 vm01 kernel: ata3: SATA link down (SStatus 0 SControl 300)
  126. Jan 25 06:13:06 vm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
  127. Jan 25 06:13:06 vm01 kernel: ata3: EH complete
  128. Jan 25 06:13:06 vm01 kernel: ata4: EH complete
  129. Jan 25 06:13:07 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  130. Jan 25 06:13:07 vm01 kernel: device-mapper: ioctl: error adding target to table
  131. Jan 25 06:13:07 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  132. Jan 25 06:13:07 vm01 kernel: device-mapper: ioctl: error adding target to table
  133. Jan 25 06:13:09 vm01 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012 File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012 s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
  134. Jan 25 06:13:09 vm01 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/import_export: rdma,port=20049
  135. Jan 25 06:13:09 vm01 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/iso: rdma,port=20049
  136. Jan 25 06:13:09 vm01 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/data: rdma,port=20049
  137. Jan 25 06:13:09 vm01 kernel: ata1: hard resetting link
  138. Jan 25 06:13:09 vm01 kernel: ata1: nv: skipping hardreset on occupied port
  139. Jan 25 06:13:09 vm01 kernel: ata2: hard resetting link
  140. Jan 25 06:13:09 vm01 kernel: ata2: nv: skipping hardreset on occupied port
  141. Jan 25 06:13:09 vm01 kernel: ata3: hard resetting link
  142. Jan 25 06:13:09 vm01 kernel: ata4: hard resetting link
  143. Jan 25 06:13:09 vm01 kernel: ata5: soft resetting link
  144. Jan 25 06:13:09 vm01 kernel: ata6: EH complete
  145. Jan 25 06:13:09 vm01 kernel: ata5: EH complete
  146. Jan 25 06:13:10 vm01 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  147. Jan 25 06:13:10 vm01 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  148. Jan 25 06:13:10 vm01 kernel: ata1.00: configured for UDMA/133
  149. Jan 25 06:13:10 vm01 kernel: ata1: EH complete
  150. Jan 25 06:13:10 vm01 kernel: ata2.00: configured for UDMA/133
  151. Jan 25 06:13:10 vm01 kernel: ata2: EH complete
  152. Jan 25 06:13:10 vm01 kernel: ata3: SATA link down (SStatus 0 SControl 300)
  153. Jan 25 06:13:10 vm01 kernel: ata3: EH complete
  154. Jan 25 06:13:10 vm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
  155. Jan 25 06:13:10 vm01 kernel: ata4: EH complete
  156. Jan 25 06:13:11 vm01 sanlock[2462]: 2014-01-25 06:13:11-0600 48136 [10244]: s3 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
  157. Jan 25 06:13:11 vm01 sanlock[2462]: 2014-01-25 06:13:11-0600 48136 [10244]: s3 renewal error -2 delta_length 10 last_success 48041
  158. Jan 25 06:13:11 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  159. Jan 25 06:13:11 vm01 kernel: device-mapper: ioctl: error adding target to table
  160. Jan 25 06:13:11 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  161. Jan 25 06:13:11 vm01 kernel: device-mapper: ioctl: error adding target to table
  162. Jan 25 06:13:12 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  163. Jan 25 06:13:13 vm01 vdsm TaskManager.Task ERROR Task=`e9436411-8bcc-4d15-8c7a-cf4bc7c26d18`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 616, in getSpmStatus#012 pool = self.getPool(spUUID)#012 File "/usr/share/vdsm/storage/hsm.py", line 318, in getPool#012 raise se.StoragePoolUnknown(spUUID)#012StoragePoolUnknown: Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)
  164. Jan 25 06:13:13 vm01 vdsm TaskManager.Task ERROR Task=`975a8818-d2ec-4821-94ed-a53784cf1e11`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2110, in getAllTasksStatuses#012 raise se.SpmStatusError()#012SpmStatusError: Not SPM: ()
  165. Jan 25 06:13:21 vm01 sanlock[2462]: 2014-01-25 06:13:21-0600 48146 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  166. Jan 25 06:13:21 vm01 sanlock[2462]: 2014-01-25 06:13:21-0600 48146 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  167. Jan 25 06:13:21 vm01 sanlock[2462]: 2014-01-25 06:13:21-0600 48146 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  168. Jan 25 06:13:21 vm01 sanlock[2462]: 2014-01-25 06:13:21-0600 48146 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  169. Jan 25 06:13:21 vm01 cpuspeed: Disabling performance cpu frequency scaling governor
  170. Jan 25 06:13:21 vm01 cpuspeed: Enabling performance cpu frequency scaling governor
  171. Jan 25 06:13:22 vm01 cpuspeed: Disabling performance cpu frequency scaling governor
  172. Jan 25 06:13:22 vm01 cpuspeed: Enabling performance cpu frequency scaling governor
  173. Jan 25 06:13:31 vm01 sanlock[2462]: 2014-01-25 06:13:31-0600 48156 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  174. Jan 25 06:13:31 vm01 sanlock[2462]: 2014-01-25 06:13:31-0600 48156 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  175. Jan 25 06:13:31 vm01 sanlock[2462]: 2014-01-25 06:13:31-0600 48156 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  176. Jan 25 06:13:31 vm01 sanlock[2462]: 2014-01-25 06:13:31-0600 48156 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  177. Jan 25 06:13:41 vm01 sanlock[2462]: 2014-01-25 06:13:41-0600 48166 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  178. Jan 25 06:13:41 vm01 sanlock[2462]: 2014-01-25 06:13:41-0600 48166 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  179. Jan 25 06:13:41 vm01 sanlock[2462]: 2014-01-25 06:13:41-0600 48166 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  180. Jan 25 06:13:41 vm01 sanlock[2462]: 2014-01-25 06:13:41-0600 48166 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  181. Jan 25 06:13:42 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  182. Jan 25 06:13:42 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  183. Jan 25 06:13:51 vm01 sanlock[2462]: 2014-01-25 06:13:51-0600 48176 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  184. Jan 25 06:13:51 vm01 sanlock[2462]: 2014-01-25 06:13:51-0600 48176 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  185. Jan 25 06:13:51 vm01 sanlock[2462]: 2014-01-25 06:13:51-0600 48176 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  186. Jan 25 06:13:51 vm01 sanlock[2462]: 2014-01-25 06:13:51-0600 48176 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  187. Jan 25 06:14:01 vm01 sanlock[2462]: 2014-01-25 06:14:01-0600 48186 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  188. Jan 25 06:14:01 vm01 sanlock[2462]: 2014-01-25 06:14:01-0600 48186 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  189. Jan 25 06:14:01 vm01 sanlock[2462]: 2014-01-25 06:14:01-0600 48186 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  190. Jan 25 06:14:01 vm01 sanlock[2462]: 2014-01-25 06:14:01-0600 48186 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  191. Jan 25 06:14:11 vm01 sanlock[2462]: 2014-01-25 06:14:11-0600 48196 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  192. Jan 25 06:14:11 vm01 sanlock[2462]: 2014-01-25 06:14:11-0600 48196 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  193. Jan 25 06:14:11 vm01 sanlock[2462]: 2014-01-25 06:14:11-0600 48196 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  194. Jan 25 06:14:11 vm01 sanlock[2462]: 2014-01-25 06:14:11-0600 48196 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  195. Jan 25 06:14:12 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  196. Jan 25 06:14:12 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  197. Jan 25 06:14:12 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  198. Jan 25 06:14:12 vm01 vdsm TaskManager.Task ERROR Task=`a703c3ac-3a0e-45e0-b15b-71265069c866`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool#012 masterVersion, options)#012 File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool#012 res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 699, in connect#012 self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild#012 masterVersion=masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain#012 raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)#012StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=7c49750d-7eae-4cd2-9b63-1dc71f357b88'
  199. Jan 25 06:14:12 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  200. Jan 25 06:14:12 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  201. Jan 25 06:14:13 vm01 vdsm TaskManager.Task ERROR Task=`7d0269cc-93b9-42ae-9afb-54771a38acae`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 616, in getSpmStatus#012 pool = self.getPool(spUUID)#012 File "/usr/share/vdsm/storage/hsm.py", line 318, in getPool#012 raise se.StoragePoolUnknown(spUUID)#012StoragePoolUnknown: Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)
  202. Jan 25 06:14:13 vm01 kernel: ata1: hard resetting link
  203. Jan 25 06:14:13 vm01 kernel: ata1: nv: skipping hardreset on occupied port
  204. Jan 25 06:14:13 vm01 kernel: ata2: hard resetting link
  205. Jan 25 06:14:13 vm01 kernel: ata2: nv: skipping hardreset on occupied port
  206. Jan 25 06:14:13 vm01 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  207. Jan 25 06:14:13 vm01 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  208. Jan 25 06:14:13 vm01 kernel: ata1.00: configured for UDMA/133
  209. Jan 25 06:14:13 vm01 kernel: ata1: EH complete
  210. Jan 25 06:14:13 vm01 kernel: ata2.00: configured for UDMA/133
  211. Jan 25 06:14:13 vm01 kernel: ata2: EH complete
  212. Jan 25 06:14:14 vm01 kernel: ata3: hard resetting link
  213. Jan 25 06:14:14 vm01 kernel: ata4: hard resetting link
  214. Jan 25 06:14:14 vm01 kernel: ata5: soft resetting link
  215. Jan 25 06:14:14 vm01 kernel: ata6: EH complete
  216. Jan 25 06:14:14 vm01 kernel: ata5: EH complete
  217. Jan 25 06:14:14 vm01 kernel: ata3: SATA link down (SStatus 0 SControl 300)
  218. Jan 25 06:14:14 vm01 kernel: ata3: EH complete
  219. Jan 25 06:14:14 vm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
  220. Jan 25 06:14:14 vm01 kernel: ata4: EH complete
  221. Jan 25 06:14:16 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  222. Jan 25 06:14:16 vm01 kernel: device-mapper: ioctl: error adding target to table
  223. Jan 25 06:14:16 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  224. Jan 25 06:14:16 vm01 kernel: device-mapper: ioctl: error adding target to table
  225. Jan 25 06:14:21 vm01 sanlock[2462]: 2014-01-25 06:14:21-0600 48206 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  226. Jan 25 06:14:21 vm01 sanlock[2462]: 2014-01-25 06:14:21-0600 48206 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  227. Jan 25 06:14:21 vm01 sanlock[2462]: 2014-01-25 06:14:21-0600 48206 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  228. Jan 25 06:14:21 vm01 sanlock[2462]: 2014-01-25 06:14:21-0600 48206 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  229. Jan 25 06:14:26 vm01 cpuspeed: Disabling performance cpu frequency scaling governor
  230. Jan 25 06:14:26 vm01 cpuspeed: Enabling performance cpu frequency scaling governor
  231. Jan 25 06:14:26 vm01 cpuspeed: Disabling performance cpu frequency scaling governor
  232. Jan 25 06:14:26 vm01 cpuspeed: Enabling performance cpu frequency scaling governor
  233. Jan 25 06:14:31 vm01 sanlock[2462]: 2014-01-25 06:14:31-0600 48216 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  234. Jan 25 06:14:31 vm01 sanlock[2462]: 2014-01-25 06:14:31-0600 48216 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  235. Jan 25 06:14:31 vm01 sanlock[2462]: 2014-01-25 06:14:31-0600 48216 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  236. Jan 25 06:14:31 vm01 sanlock[2462]: 2014-01-25 06:14:31-0600 48216 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  237. Jan 25 06:14:41 vm01 sanlock[2462]: 2014-01-25 06:14:41-0600 48226 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  238. Jan 25 06:14:41 vm01 sanlock[2462]: 2014-01-25 06:14:41-0600 48226 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  239. Jan 25 06:14:41 vm01 sanlock[2462]: 2014-01-25 06:14:41-0600 48226 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  240. Jan 25 06:14:41 vm01 sanlock[2462]: 2014-01-25 06:14:41-0600 48226 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  241. Jan 25 06:14:42 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  242. Jan 25 06:14:42 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  243. Jan 25 06:14:51 vm01 sanlock[2462]: 2014-01-25 06:14:51-0600 48236 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  244. Jan 25 06:14:51 vm01 sanlock[2462]: 2014-01-25 06:14:51-0600 48236 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  245. Jan 25 06:14:51 vm01 sanlock[2462]: 2014-01-25 06:14:51-0600 48236 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  246. Jan 25 06:14:51 vm01 sanlock[2462]: 2014-01-25 06:14:51-0600 48236 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  247. Jan 25 06:15:01 vm01 sanlock[2462]: 2014-01-25 06:15:01-0600 48246 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  248. Jan 25 06:15:01 vm01 sanlock[2462]: 2014-01-25 06:15:01-0600 48246 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  249. Jan 25 06:15:01 vm01 sanlock[2462]: 2014-01-25 06:15:01-0600 48246 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  250. Jan 25 06:15:01 vm01 sanlock[2462]: 2014-01-25 06:15:01-0600 48246 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  251. Jan 25 06:15:11 vm01 sanlock[2462]: 2014-01-25 06:15:11-0600 48256 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  252. Jan 25 06:15:11 vm01 sanlock[2462]: 2014-01-25 06:15:11-0600 48256 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  253. Jan 25 06:15:11 vm01 sanlock[2462]: 2014-01-25 06:15:11-0600 48256 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  254. Jan 25 06:15:11 vm01 sanlock[2462]: 2014-01-25 06:15:11-0600 48256 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  255. Jan 25 06:15:12 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
  256. Jan 25 06:15:12 vm01 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
  257. Jan 25 06:15:16 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  258. Jan 25 06:15:16 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  259. Jan 25 06:15:16 vm01 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012 sd.DOMAIN_META_DATA))#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012 *args, **kwargs)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012 rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012 File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012 raise Timeout()#012Timeout
  260. Jan 25 06:15:16 vm01 vdsm TaskManager.Task ERROR Task=`3fc411fc-f31e-4275-945b-53030bd23321`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool#012 masterVersion, options)#012 File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool#012 res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 699, in connect#012 self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild#012 masterVersion=masterVersion)#012 File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain#012 raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)#012StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=7c49750d-7eae-4cd2-9b63-1dc71f357b88'
  261. Jan 25 06:15:16 vm01 vdsm TaskManager.Task ERROR Task=`e4b17699-0903-4acc-9ea4-728c2d88e6b5`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2110, in getAllTasksStatuses#012 raise se.SpmStatusError()#012SpmStatusError: Not SPM: ()
  262. Jan 25 06:15:21 vm01 sanlock[2462]: 2014-01-25 06:15:21-0600 48266 [10244]: 7c49750d close_task_aio 0 0x7f37bc0008c0 busy
  263. Jan 25 06:15:21 vm01 sanlock[2462]: 2014-01-25 06:15:21-0600 48266 [10244]: 7c49750d close_task_aio 1 0x7f37bc000910 busy
  264. Jan 25 06:15:21 vm01 sanlock[2462]: 2014-01-25 06:15:21-0600 48266 [10244]: 7c49750d close_task_aio 2 0x7f37bc000960 busy
  265. Jan 25 06:15:21 vm01 sanlock[2462]: 2014-01-25 06:15:21-0600 48266 [10244]: 7c49750d close_task_aio 3 0x7f37bc0009b0 busy
  266. Jan 25 06:15:26 vm01 vdsm TaskManager.Task ERROR Task=`a3b4b543-fb23-42bb-a478-4cd17b503247`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 857, in _run#012 return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 616, in getSpmStatus#012 pool = self.getPool(spUUID)#012 File "/usr/share/vdsm/storage/hsm.py", line 318, in getPool#012 raise se.StoragePoolUnknown(spUUID)#012StoragePoolUnknown: Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)
  267. Jan 25 06:15:26 vm01 kernel: ata3: hard resetting link
  268. Jan 25 06:15:26 vm01 kernel: ata2: hard resetting link
  269. Jan 25 06:15:26 vm01 kernel: ata2: nv: skipping hardreset on occupied port
  270. Jan 25 06:15:26 vm01 kernel: ata4: hard resetting link
  271. Jan 25 06:15:26 vm01 kernel: ata1: hard resetting link
  272. Jan 25 06:15:26 vm01 kernel: ata1: nv: skipping hardreset on occupied port
  273. Jan 25 06:15:26 vm01 kernel: ata5: soft resetting link
  274. Jan 25 06:15:26 vm01 kernel: ata6: EH complete
  275. Jan 25 06:15:26 vm01 kernel: ata5: EH complete
  276. Jan 25 06:15:27 vm01 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  277. Jan 25 06:15:27 vm01 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
  278. Jan 25 06:15:27 vm01 kernel: ata2.00: configured for UDMA/133
  279. Jan 25 06:15:27 vm01 kernel: ata2: EH complete
  280. Jan 25 06:15:27 vm01 kernel: ata1.00: configured for UDMA/133
  281. Jan 25 06:15:27 vm01 kernel: ata1: EH complete
  282. Jan 25 06:15:27 vm01 kernel: ata3: SATA link down (SStatus 0 SControl 300)
  283. Jan 25 06:15:27 vm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
  284. Jan 25 06:15:27 vm01 kernel: ata3: EH complete
  285. Jan 25 06:15:27 vm01 kernel: ata4: EH complete
  286. Jan 25 06:15:28 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  287. Jan 25 06:15:28 vm01 kernel: device-mapper: ioctl: error adding target to table
  288. Jan 25 06:15:28 vm01 kernel: device-mapper: table: 253:2: multipath: error getting device
  289. Jan 25 06:15:28 vm01 kernel: device-mapper: ioctl: error adding target to table
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!