API tools faq
paste
cweise

OVS Crash & Burn

cweise
Jun 13th, 2012
256
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 103.28 KB | None | 0 0
raw download clone embed print report
  1. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: I/O queue driver: lio
  2. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: tapdisk-log: started, level 0
  3. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd version: tap 0x00010003, b: 153600, a: 0, f: 0, n: 124
  4. 2
  5. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: opened image /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd (1 users, state: 0x00000001, type: 4, rw)
  6.  
  7. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: VBD CHAIN:
  8. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: type:vhd(4) storage:nfs(1)
  9. Jun 13 01:04:46 AGC-A2X2 tapdisk[15206]: bdev: capacity=629145600 sector_size=512/512 flags=0
  10. Jun 13 01:04:46 AGC-A2X2 kernel: block tdh: sector-size: 512/512 capacity: 629145600
  11. Jun 13 01:04:46 AGC-A2X2 kernel: blkback: event-channel 51
  12. Jun 13 01:04:46 AGC-A2X2 kernel: blkback: ring-ref 219
  13. Jun 13 01:04:46 AGC-A2X2 kernel: blkback: protocol 1 (x86_32-abi)
  14. Jun 13 01:04:46 AGC-A2X2 kernel: blkfront: xvda: barriers enabled
  15. Jun 13 01:04:46 AGC-A2X2 kernel: xvda: unknown partition table
  16. Jun 13 01:04:47 AGC-A2X2 vbd.uevent[add](backend/vbd/0/51712): wrote /xapi/0/hotplug/vbd/51712/hotplug = 'online'
  17. Jun 13 01:04:47 AGC-A2X2 vbd.uevent[add](backend/vbd/0/51712): wrote backend/vbd/0/51712/hotplug-status = 'connected'
  18. Jun 13 01:05:10 AGC-A2X2 ovsdb-server: 00068|timeval|WARN|358 ms poll interval (10 ms user, 0 ms system) is over 477 times the weighted mean interval 1 ms (36027 samples)
  19. Jun 13 01:05:10 AGC-A2X2 ovsdb-server: 00069|timeval|WARN|context switches: 7 voluntary, 0 involuntary
  20. Jun 13 01:05:10 AGC-A2X2 ovsdb-server: 00070|coverage|INFO|Skipping details of duplicate event coverage for hash=abb17da2 in epoch 36027
  21. Jun 13 01:06:35 AGC-A2X2 ovsdb-server: 00071|timeval|WARN|35 ms poll interval (0 ms user, 10 ms system) is over 14 times the weighted mean interval 3 ms (36076 samples)
  22. Jun 13 01:06:35 AGC-A2X2 ovsdb-server: 00072|timeval|WARN|faults: 1 minor, 0 major
  23. Jun 13 01:06:35 AGC-A2X2 ovsdb-server: 00073|timeval|WARN|context switches: 2 voluntary, 0 involuntary
  24. Jun 13 01:06:35 AGC-A2X2 ovsdb-server: 00074|coverage|INFO|Skipping details of duplicate event coverage for hash=abb17da2 in epoch 36076
  25. Jun 13 01:06:55 AGC-A2X2 ovs-vswitchd: 02091|timeval|WARN|11 ms poll interval (0 ms user, 10 ms system) is over 8 times the weighted mean interval 1 ms (1301851 samples)
  26. Jun 13 01:06:55 AGC-A2X2 ovs-vswitchd: 02092|timeval|WARN|context switches: 2 voluntary, 0 involuntary
  27. Jun 13 01:06:55 AGC-A2X2 ovs-vswitchd: 02093|coverage|INFO|Skipping details of duplicate event coverage for hash=a801370e in epoch 1301851
  28. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00075|reconnect|ERR|unix:/tmp/stream-unix.6488.0: no response to inactivity probe after 5.01 seconds, disconnecting
  29. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00076|timeval|WARN|8 ms poll interval (0 ms user, 0 ms system) is over 12 times the weighted mean interval 1 ms (36162 samples)
  30. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00077|timeval|WARN|faults: 5 minor, 1 major
  31. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00078|timeval|WARN|context switches: 1 voluntary, 0 involuntary
  32. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00079|coverage|INFO|Event coverage (epoch 36162/entire run), hash=e768c78c:
  33. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00080|coverage|INFO|poll_fd_wait 3 / 135962
  34. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00081|coverage|INFO|util_xalloc 19 / 73227386
  35. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00082|coverage|INFO|hmap_pathological 0 / 663
  36. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00083|coverage|INFO|hmap_expand 0 / 848706
  37. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00084|coverage|INFO|lockfile_lock 0 / 1
  38. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00085|coverage|INFO|poll_zero_timeout 0 / 9139
  39. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00086|coverage|INFO|pstream_open 0 / 1
  40. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00087|coverage|INFO|unixctl_received 0 / 1
  41. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00088|coverage|INFO|unixctl_replied 0 / 1
  42. Jun 13 01:09:05 AGC-A2X2 ovsdb-server: 00089|coverage|INFO|10 events never hit
  43. Jun 13 01:09:05 AGC-A2X2 tapdisk[15206]: logfile saved to /var/log/blktap/tapdisk.15206.log: 0
  44. Jun 13 01:09:09 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3605800
  45. , secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  46. Jun 13 01:09:09 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3605888
  47. , secs: 32, nbytes: 16384, blk: 880, blk_offset: 3612767
  48. Jun 13 01:09:09 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3616272
  49. , secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  50. Jun 13 01:09:09 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 5, lsec: 3612672
  51. , secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  52. Jun 13 01:09:09 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.0: write 0x0008 secs @ 0x00372e10 - Input/output error
  53. Jun 13 01:09:09 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  54. Jun 13 01:09:09 AGC-A2X2 last message repeated 3 times
  55. Jun 13 01:09:28 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16640 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  56. Jun 13 01:09:48 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 4, lsec: 3604480, secs: 1, nbytes: 512, blk: 880, blk_offset: 3612767
  57. Jun 13 01:09:48 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.1: write 0x0058 secs @ 0x00370528 - Input/output error
  58. Jun 13 01:09:48 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.2: write 0x0020 secs @ 0x00370580 - Input/output error
  59. Jun 13 01:09:48 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  60. Jun 13 01:10:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 5, lsec: 3612672, secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  61. Jun 13 01:10:03 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  62. Jun 13 01:10:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16654 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  63. Jun 13 01:10:39 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3616272, secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  64. Jun 13 01:10:39 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  65. Jun 13 01:10:49 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16685 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  66. Jun 13 01:11:14 AGC-A2X2 kernel: INFO: task ovs-vswitchd:6488 blocked for more than 120 seconds.
  67. Jun 13 01:11:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  68. Jun 13 01:11:14 AGC-A2X2 kernel: ovs-vswitchd D eeb1c2c0 0 6488 6487 0x00000004
  69. Jun 13 01:11:14 AGC-A2X2 kernel: e4f439dc 00200286 00000017 eeb1c2c0 00000010 e6937e0c 0000000f eeb1c2c0
  70. Jun 13 01:11:14 AGC-A2X2 kernel: 00517556 0000282f ed7869d4 ed7868c4 ed786830 ed7869d4 c16cc100 00000001
  71. Jun 13 01:11:14 AGC-A2X2 kernel: e614c200 0000282f c04cb780 000067fd 00000000 00000005 0083554f 3931964e
  72. Jun 13 01:11:14 AGC-A2X2 kernel: Call Trace:
  73. Jun 13 01:11:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  74. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0163145>] sync_page+0x35/0x40
  75. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  76. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0163110>] ? sync_page+0x0/0x40
  77. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0163515>] wait_on_page_bit+0x55/0x60
  78. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  79. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016e992>] shrink_page_list+0x662/0x690
  80. Jun 13 01:11:14 AGC-A2X2 kernel: [<c015fdf6>] ? delayacct_end+0x86/0xa0
  81. Jun 13 01:11:14 AGC-A2X2 kernel: [<c015ff6c>] ? __delayacct_blkio_end+0x6c/0x80
  82. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016f0c2>] shrink_list+0x702/0x7e0
  83. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016a54f>] ? determine_dirtyable_memory+0x3f/0x60
  84. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016a585>] ? get_dirty_limits+0x15/0x2d0
  85. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016f534>] shrink_zone+0x274/0x360
  86. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016fa51>] try_to_free_pages+0x1a1/0x2b0
  87. Jun 13 01:11:14 AGC-A2X2 kernel: [<c016d520>] ? isolate_pages_global+0x0/0x1d0
  88. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0169028>] __alloc_pages_nodemask+0x238/0x4f0
  89. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0120e48>] ? __wake_up_common+0x48/0x70
  90. Jun 13 01:11:14 AGC-A2X2 kernel: [<c018f9d2>] cache_alloc_refill+0x312/0x610
  91. Jun 13 01:11:14 AGC-A2X2 kernel: [<c018fe23>] __kmalloc+0x153/0x180
  92. Jun 13 01:11:14 AGC-A2X2 kernel: [<c030014f>] __alloc_skb+0x6f/0x170
  93. Jun 13 01:11:14 AGC-A2X2 kernel: [<c020257f>] ? security_sock_rcv_skb+0xf/0x20
  94. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02fb1cd>] ? sock_alloc_send_pskb+0x15d/0x280
  95. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02fb1cd>] sock_alloc_send_pskb+0x15d/0x280
  96. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02fb308>] sock_alloc_send_skb+0x18/0x20
  97. Jun 13 01:11:14 AGC-A2X2 kernel: [<c036ccd4>] unix_stream_sendmsg+0x1c4/0x350
  98. Jun 13 01:11:14 AGC-A2X2 kernel: [<c032ac12>] ? ip_rcv+0x1e2/0x2c0
  99. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02f7f3a>] sock_aio_write+0x10a/0x120
  100. Jun 13 01:11:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  101. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  102. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  103. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  104. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0196b56>] vfs_write+0x146/0x150
  105. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02f9bfc>] ? sys_socketcall+0x25c/0x270
  106. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  107. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  108. Jun 13 01:11:14 AGC-A2X2 kernel: INFO: task xapi:14978 blocked for more than 120 seconds.
  109. Jun 13 01:11:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  110. Jun 13 01:11:14 AGC-A2X2 kernel: xapi D f4d62ddc 0 14978 9257 0x00000000
  111. Jun 13 01:11:14 AGC-A2X2 kernel: ecd1fc80 00000282 afdff2ba f4d62ddc 0000282e 00000005 00374080 00000000
  112. Jun 13 01:11:14 AGC-A2X2 kernel: e0256bac 00374080 eced49d4 eced48c4 eced4830 eced49d4 c16cc100 00000001
  113. Jun 13 01:11:14 AGC-A2X2 kernel: e61d8740 0000282f c04cb780 00000000 00000000 ecd1fc78 c014a3a6 3b60c865
  114. Jun 13 01:11:14 AGC-A2X2 kernel: Call Trace:
  115. Jun 13 01:11:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  116. Jun 13 01:11:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  117. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b9035>] sync_buffer+0x35/0x40
  118. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  119. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  120. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  121. Jun 13 01:11:14 AGC-A2X2 kernel: [<c03837f1>] out_of_line_wait_on_bit+0x51/0x60
  122. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  123. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b8f86>] __wait_on_buffer+0x26/0x30
  124. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b9753>] __block_prepare_write+0x193/0x380
  125. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01b99c8>] block_write_begin+0x48/0xe0
  126. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  127. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01be4e3>] blkdev_write_begin+0x43/0x50
  128. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  129. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0163d22>] generic_file_buffered_write+0xd2/0x240
  130. Jun 13 01:11:14 AGC-A2X2 kernel: [<c011179e>] ? xen_smp_send_reschedule+0x2e/0x50
  131. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0122306>] ? resched_task+0x56/0x60
  132. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01642fc>] __generic_file_aio_write+0x26c/0x540
  133. Jun 13 01:11:14 AGC-A2X2 kernel: [<c02f8065>] ? sock_aio_read+0x115/0x120
  134. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0106bb1>] ? local_clock+0x41/0x80
  135. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01bdae8>] blkdev_aio_write+0x38/0x90
  136. Jun 13 01:11:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  137. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  138. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  139. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  140. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0196aa6>] vfs_write+0x96/0x150
  141. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0196250>] ? do_sync_write+0x0/0x110
  142. Jun 13 01:11:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  143. Jun 13 01:11:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  144. Jun 13 01:11:27 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  145. Jun 13 01:11:29 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16698 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  146. Jun 13 01:12:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16711 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  147. Jun 13 01:12:27 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3605800, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  148. Jun 13 01:12:27 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3605888, secs: 32, nbytes: 16384, blk: 880, blk_offset: 3612767
  149. Jun 13 01:12:27 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  150. Jun 13 01:12:49 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16729 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  151. Jun 13 01:13:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 5, lsec: 3612672, secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  152. Jun 13 01:13:03 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  153. Jun 13 01:13:14 AGC-A2X2 kernel: INFO: task ovs-vswitchd:6488 blocked for more than 120 seconds.
  154. Jun 13 01:13:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  155. Jun 13 01:13:14 AGC-A2X2 kernel: ovs-vswitchd D eeb1c2c0 0 6488 6487 0x00000004
  156. Jun 13 01:13:14 AGC-A2X2 kernel: e4f439dc 00200286 00000017 eeb1c2c0 00000010 e6937e0c 0000000f eeb1c2c0
  157. Jun 13 01:13:14 AGC-A2X2 kernel: 00517556 0000282f ed7869d4 ed7868c4 ed786830 ed7869d4 c16cc100 00000001
  158. Jun 13 01:13:14 AGC-A2X2 kernel: e614c200 0000282f c04cb780 000067fd 00000000 00000005 0083554f 3931964e
  159. Jun 13 01:13:14 AGC-A2X2 kernel: Call Trace:
  160. Jun 13 01:13:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  161. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0163145>] sync_page+0x35/0x40
  162. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  163. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0163110>] ? sync_page+0x0/0x40
  164. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0163515>] wait_on_page_bit+0x55/0x60
  165. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  166. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016e992>] shrink_page_list+0x662/0x690
  167. Jun 13 01:13:14 AGC-A2X2 kernel: [<c015fdf6>] ? delayacct_end+0x86/0xa0
  168. Jun 13 01:13:14 AGC-A2X2 kernel: [<c015ff6c>] ? __delayacct_blkio_end+0x6c/0x80
  169. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016f0c2>] shrink_list+0x702/0x7e0
  170. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a54f>] ? determine_dirtyable_memory+0x3f/0x60
  171. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a585>] ? get_dirty_limits+0x15/0x2d0
  172. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016f534>] shrink_zone+0x274/0x360
  173. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016fa51>] try_to_free_pages+0x1a1/0x2b0
  174. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016d520>] ? isolate_pages_global+0x0/0x1d0
  175. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0169028>] __alloc_pages_nodemask+0x238/0x4f0
  176. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0120e48>] ? __wake_up_common+0x48/0x70
  177. Jun 13 01:13:14 AGC-A2X2 kernel: [<c018f9d2>] cache_alloc_refill+0x312/0x610
  178. Jun 13 01:13:14 AGC-A2X2 kernel: [<c018fe23>] __kmalloc+0x153/0x180
  179. Jun 13 01:13:14 AGC-A2X2 kernel: [<c030014f>] __alloc_skb+0x6f/0x170
  180. Jun 13 01:13:14 AGC-A2X2 kernel: [<c020257f>] ? security_sock_rcv_skb+0xf/0x20
  181. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02fb1cd>] ? sock_alloc_send_pskb+0x15d/0x280
  182. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02fb1cd>] sock_alloc_send_pskb+0x15d/0x280
  183. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02fb308>] sock_alloc_send_skb+0x18/0x20
  184. Jun 13 01:13:14 AGC-A2X2 kernel: [<c036ccd4>] unix_stream_sendmsg+0x1c4/0x350
  185. Jun 13 01:13:14 AGC-A2X2 kernel: [<c032ac12>] ? ip_rcv+0x1e2/0x2c0
  186. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02f7f3a>] sock_aio_write+0x10a/0x120
  187. Jun 13 01:13:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  188. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  189. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  190. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  191. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0196b56>] vfs_write+0x146/0x150
  192. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02f9bfc>] ? sys_socketcall+0x25c/0x270
  193. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  194. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  195. Jun 13 01:13:14 AGC-A2X2 kernel: INFO: task xapi:14978 blocked for more than 120 seconds.
  196. Jun 13 01:13:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  197. Jun 13 01:13:14 AGC-A2X2 kernel: xapi D f4d62ddc 0 14978 9257 0x00000000
  198. Jun 13 01:13:14 AGC-A2X2 kernel: ecd1fc80 00000282 afdff2ba f4d62ddc 0000282e 00000005 00374080 00000000
  199. Jun 13 01:13:14 AGC-A2X2 kernel: e0256bac 00374080 eced49d4 eced48c4 eced4830 eced49d4 c16cc100 00000001
  200. Jun 13 01:13:14 AGC-A2X2 kernel: e61d8740 0000282f c04cb780 00000000 00000000 ecd1fc78 c014a3a6 3b60c865
  201. Jun 13 01:13:14 AGC-A2X2 kernel: Call Trace:
  202. Jun 13 01:13:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  203. Jun 13 01:13:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  204. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b9035>] sync_buffer+0x35/0x40
  205. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  206. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  207. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  208. Jun 13 01:13:14 AGC-A2X2 kernel: [<c03837f1>] out_of_line_wait_on_bit+0x51/0x60
  209. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  210. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b8f86>] __wait_on_buffer+0x26/0x30
  211. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b9753>] __block_prepare_write+0x193/0x380
  212. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b99c8>] block_write_begin+0x48/0xe0
  213. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  214. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01be4e3>] blkdev_write_begin+0x43/0x50
  215. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  216. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0163d22>] generic_file_buffered_write+0xd2/0x240
  217. Jun 13 01:13:14 AGC-A2X2 kernel: [<c011179e>] ? xen_smp_send_reschedule+0x2e/0x50
  218. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0122306>] ? resched_task+0x56/0x60
  219. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01642fc>] __generic_file_aio_write+0x26c/0x540
  220. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02f8065>] ? sock_aio_read+0x115/0x120
  221. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0106bb1>] ? local_clock+0x41/0x80
  222. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bdae8>] blkdev_aio_write+0x38/0x90
  223. Jun 13 01:13:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  224. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  225. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  226. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  227. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0196aa6>] vfs_write+0x96/0x150
  228. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0196250>] ? do_sync_write+0x0/0x110
  229. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  230. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  231. Jun 13 01:13:14 AGC-A2X2 kernel: INFO: task flush-202:0:15253 blocked for more than 120 seconds.
  232. Jun 13 01:13:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  233. Jun 13 01:13:14 AGC-A2X2 kernel: flush-202:0 D e9f03be0 0 15253 2 0x00000000
  234. Jun 13 01:13:14 AGC-A2X2 kernel: e9f03bf4 00000246 00000002 e9f03be0 c16eb224 00000000 00000001 00000000
  235. Jun 13 01:13:14 AGC-A2X2 kernel: 00000000 e0256a18 e5b7e994 e5b7e884 e5b7e7f0 e5b7e994 c16ec100 00000003
  236. Jun 13 01:13:14 AGC-A2X2 kernel: e624d580 0042edaf 00000000 00000000 00000000 e9f03bec c014a3a6 399b1a1e
  237. Jun 13 01:13:14 AGC-A2X2 kernel: Call Trace:
  238. Jun 13 01:13:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  239. Jun 13 01:13:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  240. Jun 13 01:13:14 AGC-A2X2 kernel: [<c021375f>] get_request_wait+0x5f/0x120
  241. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  242. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0213c6a>] __make_request+0x7a/0x3c0
  243. Jun 13 01:13:14 AGC-A2X2 kernel: [<c018f72c>] ? cache_alloc_refill+0x6c/0x610
  244. Jun 13 01:13:14 AGC-A2X2 kernel: [<c02123d6>] generic_make_request+0xf6/0x290
  245. Jun 13 01:13:14 AGC-A2X2 kernel: [<c018fe72>] ? kmem_cache_alloc+0x22/0x130
  246. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016582e>] ? mempool_alloc_slab+0xe/0x10
  247. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0165ab7>] ? mempool_alloc+0x37/0x110
  248. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0213ade>] submit_bio+0x4e/0xd0
  249. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bc3b0>] ? bio_alloc_bioset+0x40/0xc0
  250. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b7f94>] submit_bh+0xe4/0x140
  251. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b9fb0>] __block_write_full_page+0x200/0x370
  252. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  253. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01ba1fd>] block_write_full_page_endio+0xdd/0x120
  254. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  255. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  256. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01ba252>] block_write_full_page+0x12/0x20
  257. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  258. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01be51f>] blkdev_writepage+0xf/0x20
  259. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0169e1b>] __writepage+0xb/0x30
  260. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a1bd>] write_cache_pages+0x1fd/0x3c0
  261. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0169e10>] ? __writepage+0x0/0x30
  262. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a3a3>] generic_writepages+0x23/0x30
  263. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a380>] ? generic_writepages+0x0/0x30
  264. Jun 13 01:13:14 AGC-A2X2 kernel: [<c016a3cd>] do_writepages+0x1d/0x30
  265. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b29f9>] writeback_single_inode+0x99/0x270
  266. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b347e>] writeback_inodes_wb+0x2ae/0x420
  267. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b36ec>] wb_writeback+0xfc/0x1b0
  268. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b39c8>] wb_do_writeback+0x158/0x180
  269. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01b3a15>] bdi_writeback_task+0x25/0xa0
  270. Jun 13 01:13:14 AGC-A2X2 kernel: [<c017603e>] bdi_start_fn+0x5e/0xd0
  271. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0175fe0>] ? bdi_start_fn+0x0/0xd0
  272. Jun 13 01:13:14 AGC-A2X2 kernel: [<c0142854>] kthread+0x74/0x80
  273. Jun 13 01:13:14 AGC-A2X2 kernel: [<c01427e0>] ? kthread+0x0/0x80
  274. Jun 13 01:13:14 AGC-A2X2 kernel: [<c010483b>] kernel_thread_helper+0x7/0x10
  275. Jun 13 01:13:29 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16742 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  276. Jun 13 01:14:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3616272, secs: 8, nbytes: 4096, blk: 882, blk_offset: 4294967295
  277. Jun 13 01:14:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.0 timed out, retried 2 times
  278. Jun 13 01:14:03 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  279. Jun 13 01:14:03 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3616272
  280. Jun 13 01:14:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3616272
  281. Jun 13 01:14:03 AGC-A2X2 kernel: __ratelimit: 2 callbacks suppressed
  282. Jun 13 01:14:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 452034
  283. Jun 13 01:14:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  284. Jun 13 01:14:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16755 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  285. Jun 13 01:14:49 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16773 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  286. Jun 13 01:15:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 4, lsec: 3604480, secs: 1, nbytes: 512, blk: 880, blk_offset: 3612767
  287. Jun 13 01:15:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.1 timed out, retried 1 times
  288. Jun 13 01:15:03 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.2 timed out, retried 1 times
  289. Jun 13 01:15:03 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  290. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3605800
  291. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605800
  292. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450725
  293. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  294. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605808
  295. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450726
  296. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  297. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605816
  298. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450727
  299. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  300. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605824
  301. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450728
  302. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  303. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605832
  304. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450729
  305. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  306. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605840
  307. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450730
  308. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  309. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605848
  310. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450731
  311. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  312. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605856
  313. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450732
  314. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  315. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605864
  316. Jun 13 01:15:03 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450733
  317. Jun 13 01:15:03 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  318. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605872
  319. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605880
  320. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3605888
  321. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605888
  322. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605896
  323. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605904
  324. Jun 13 01:15:03 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605912
  325. Jun 13 01:15:14 AGC-A2X2 kernel: INFO: task flush-202:0:15253 blocked for more than 120 seconds.
  326. Jun 13 01:15:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  327. Jun 13 01:15:14 AGC-A2X2 kernel: flush-202:0 D e9f03be0 0 15253 2 0x00000000
  328. Jun 13 01:15:14 AGC-A2X2 kernel: e9f03bf4 00000246 00000002 e9f03be0 c16eb224 00000000 00000001 00000000
  329. Jun 13 01:15:14 AGC-A2X2 kernel: 00000000 e0256a18 e5b7e994 e5b7e884 e5b7e7f0 e5b7e994 c16ec100 00000003
  330. Jun 13 01:15:14 AGC-A2X2 kernel: e624d580 0042edaf 00000000 00000000 00000000 e9f03bec c014a3a6 399b1a1e
  331. Jun 13 01:15:14 AGC-A2X2 kernel: Call Trace:
  332. Jun 13 01:15:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  333. Jun 13 01:15:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  334. Jun 13 01:15:14 AGC-A2X2 kernel: [<c021375f>] get_request_wait+0x5f/0x120
  335. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  336. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0213c6a>] __make_request+0x7a/0x3c0
  337. Jun 13 01:15:14 AGC-A2X2 kernel: [<c018f72c>] ? cache_alloc_refill+0x6c/0x610
  338. Jun 13 01:15:14 AGC-A2X2 kernel: [<c02123d6>] generic_make_request+0xf6/0x290
  339. Jun 13 01:15:14 AGC-A2X2 kernel: [<c018fe72>] ? kmem_cache_alloc+0x22/0x130
  340. Jun 13 01:15:14 AGC-A2X2 kernel: [<c016582e>] ? mempool_alloc_slab+0xe/0x10
  341. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0165ab7>] ? mempool_alloc+0x37/0x110
  342. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0213ade>] submit_bio+0x4e/0xd0
  343. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01bc3b0>] ? bio_alloc_bioset+0x40/0xc0
  344. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b7f94>] submit_bh+0xe4/0x140
  345. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b9fb0>] __block_write_full_page+0x200/0x370
  346. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  347. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01ba1fd>] block_write_full_page_endio+0xdd/0x120
  348. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  349. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  350. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01ba252>] block_write_full_page+0x12/0x20
  351. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  352. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01be51f>] blkdev_writepage+0xf/0x20
  353. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0169e1b>] __writepage+0xb/0x30
  354. Jun 13 01:15:14 AGC-A2X2 kernel: [<c016a1bd>] write_cache_pages+0x1fd/0x3c0
  355. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0169e10>] ? __writepage+0x0/0x30
  356. Jun 13 01:15:14 AGC-A2X2 kernel: [<c016a3a3>] generic_writepages+0x23/0x30
  357. Jun 13 01:15:14 AGC-A2X2 kernel: [<c016a380>] ? generic_writepages+0x0/0x30
  358. Jun 13 01:15:14 AGC-A2X2 kernel: [<c016a3cd>] do_writepages+0x1d/0x30
  359. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b29f9>] writeback_single_inode+0x99/0x270
  360. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b347e>] writeback_inodes_wb+0x2ae/0x420
  361. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b36ec>] wb_writeback+0xfc/0x1b0
  362. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b39c8>] wb_do_writeback+0x158/0x180
  363. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01b3a15>] bdi_writeback_task+0x25/0xa0
  364. Jun 13 01:15:14 AGC-A2X2 kernel: [<c017603e>] bdi_start_fn+0x5e/0xd0
  365. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0175fe0>] ? bdi_start_fn+0x0/0xd0
  366. Jun 13 01:15:14 AGC-A2X2 kernel: [<c0142854>] kthread+0x74/0x80
  367. Jun 13 01:15:14 AGC-A2X2 kernel: [<c01427e0>] ? kthread+0x0/0x80
  368. Jun 13 01:15:14 AGC-A2X2 kernel: [<c010483b>] kernel_thread_helper+0x7/0x10
  369. Jun 13 01:15:29 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16799 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  370. Jun 13 01:16:03 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  371. Jun 13 01:16:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16813 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  372. Jun 13 01:16:49 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16831 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  373. Jun 13 01:17:14 AGC-A2X2 kernel: INFO: task ovs-vswitchd:6488 blocked for more than 120 seconds.
  374. Jun 13 01:17:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  375. Jun 13 01:17:14 AGC-A2X2 kernel: ovs-vswitchd D e4f439c8 0 6488 6487 0x00000004
  376. Jun 13 01:17:14 AGC-A2X2 kernel: e4f439dc 00200286 00000002 e4f439c8 c16eb224 00000000 c04cb780 00000000
  377. Jun 13 01:17:14 AGC-A2X2 kernel: 00000000 e4f439a4 ed7869d4 ed7868c4 ed786830 ed7869d4 c16ec100 00000003
  378. Jun 13 01:17:14 AGC-A2X2 kernel: e614c200 00435dea 00000000 00000000 00000000 e4f439d4 c014a3a6 0f390cfd
  379. Jun 13 01:17:14 AGC-A2X2 kernel: Call Trace:
  380. Jun 13 01:17:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  381. Jun 13 01:17:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  382. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0163145>] sync_page+0x35/0x40
  383. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  384. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0163110>] ? sync_page+0x0/0x40
  385. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0163515>] wait_on_page_bit+0x55/0x60
  386. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  387. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016e793>] shrink_page_list+0x463/0x690
  388. Jun 13 01:17:14 AGC-A2X2 kernel: [<c015fdf6>] ? delayacct_end+0x86/0xa0
  389. Jun 13 01:17:14 AGC-A2X2 kernel: [<c015ff6c>] ? __delayacct_blkio_end+0x6c/0x80
  390. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016f0c2>] shrink_list+0x702/0x7e0
  391. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a54f>] ? determine_dirtyable_memory+0x3f/0x60
  392. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a585>] ? get_dirty_limits+0x15/0x2d0
  393. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016f534>] shrink_zone+0x274/0x360
  394. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016fa51>] try_to_free_pages+0x1a1/0x2b0
  395. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016d520>] ? isolate_pages_global+0x0/0x1d0
  396. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0169028>] __alloc_pages_nodemask+0x238/0x4f0
  397. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0120e48>] ? __wake_up_common+0x48/0x70
  398. Jun 13 01:17:14 AGC-A2X2 kernel: [<c018f9d2>] cache_alloc_refill+0x312/0x610
  399. Jun 13 01:17:14 AGC-A2X2 kernel: [<c018fe23>] __kmalloc+0x153/0x180
  400. Jun 13 01:17:14 AGC-A2X2 kernel: [<c030014f>] __alloc_skb+0x6f/0x170
  401. Jun 13 01:17:14 AGC-A2X2 kernel: [<c020257f>] ? security_sock_rcv_skb+0xf/0x20
  402. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02fb1cd>] ? sock_alloc_send_pskb+0x15d/0x280
  403. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02fb1cd>] sock_alloc_send_pskb+0x15d/0x280
  404. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02fb308>] sock_alloc_send_skb+0x18/0x20
  405. Jun 13 01:17:14 AGC-A2X2 kernel: [<c036ccd4>] unix_stream_sendmsg+0x1c4/0x350
  406. Jun 13 01:17:14 AGC-A2X2 kernel: [<c032ac12>] ? ip_rcv+0x1e2/0x2c0
  407. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02f7f3a>] sock_aio_write+0x10a/0x120
  408. Jun 13 01:17:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  409. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  410. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  411. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  412. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0196b56>] vfs_write+0x146/0x150
  413. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02f9bfc>] ? sys_socketcall+0x25c/0x270
  414. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  415. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  416. Jun 13 01:17:14 AGC-A2X2 kernel: INFO: task xapi:14978 blocked for more than 120 seconds.
  417. Jun 13 01:17:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  418. Jun 13 01:17:14 AGC-A2X2 kernel: xapi D 00002884 0 14978 9257 0x00000000
  419. Jun 13 01:17:14 AGC-A2X2 kernel: ecd1fc80 00000282 8637f78f 00002884 8637f78f 00002884 003740a0 00000000
  420. Jun 13 01:17:14 AGC-A2X2 kernel: 8638d2a3 00002884 eced49d4 eced48c4 eced4830 eced49d4 c16cc100 00000001
  421. Jun 13 01:17:14 AGC-A2X2 kernel: e61d8740 00002884 c04cb780 00001b7d 00000000 ecd1fc78 c014a3a6 0ef9add9
  422. Jun 13 01:17:14 AGC-A2X2 kernel: Call Trace:
  423. Jun 13 01:17:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  424. Jun 13 01:17:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  425. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b9035>] sync_buffer+0x35/0x40
  426. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  427. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  428. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b9000>] ? sync_buffer+0x0/0x40
  429. Jun 13 01:17:14 AGC-A2X2 kernel: [<c03837f1>] out_of_line_wait_on_bit+0x51/0x60
  430. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  431. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b8f86>] __wait_on_buffer+0x26/0x30
  432. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b9753>] __block_prepare_write+0x193/0x380
  433. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b99c8>] block_write_begin+0x48/0xe0
  434. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  435. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01be4e3>] blkdev_write_begin+0x43/0x50
  436. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  437. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0163d22>] generic_file_buffered_write+0xd2/0x240
  438. Jun 13 01:17:14 AGC-A2X2 kernel: [<c011179e>] ? xen_smp_send_reschedule+0x2e/0x50
  439. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0122306>] ? resched_task+0x56/0x60
  440. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01642fc>] __generic_file_aio_write+0x26c/0x540
  441. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02f8065>] ? sock_aio_read+0x115/0x120
  442. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0106bb1>] ? local_clock+0x41/0x80
  443. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bdae8>] blkdev_aio_write+0x38/0x90
  444. Jun 13 01:17:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  445. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  446. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  447. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  448. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0196aa6>] vfs_write+0x96/0x150
  449. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0196250>] ? do_sync_write+0x0/0x110
  450. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  451. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  452. Jun 13 01:17:14 AGC-A2X2 kernel: INFO: task flush-202:0:15253 blocked for more than 120 seconds.
  453. Jun 13 01:17:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  454. Jun 13 01:17:14 AGC-A2X2 kernel: flush-202:0 D e9f03be0 0 15253 2 0x00000000
  455. Jun 13 01:17:14 AGC-A2X2 kernel: e9f03bf4 00000246 00000002 e9f03be0 c16eb224 00000000 00000001 00000000
  456. Jun 13 01:17:14 AGC-A2X2 kernel: 00000000 e0256a18 e5b7e994 e5b7e884 e5b7e7f0 e5b7e994 c16ec100 00000003
  457. Jun 13 01:17:14 AGC-A2X2 kernel: e624d580 0042edaf 00000000 00000000 00000000 e9f03bec c014a3a6 399b1a1e
  458. Jun 13 01:17:14 AGC-A2X2 kernel: Call Trace:
  459. Jun 13 01:17:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  460. Jun 13 01:17:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  461. Jun 13 01:17:14 AGC-A2X2 kernel: [<c021375f>] get_request_wait+0x5f/0x120
  462. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  463. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0213c6a>] __make_request+0x7a/0x3c0
  464. Jun 13 01:17:14 AGC-A2X2 kernel: [<c018f72c>] ? cache_alloc_refill+0x6c/0x610
  465. Jun 13 01:17:14 AGC-A2X2 kernel: [<c02123d6>] generic_make_request+0xf6/0x290
  466. Jun 13 01:17:14 AGC-A2X2 kernel: [<c018fe72>] ? kmem_cache_alloc+0x22/0x130
  467. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016582e>] ? mempool_alloc_slab+0xe/0x10
  468. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0165ab7>] ? mempool_alloc+0x37/0x110
  469. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0213ade>] submit_bio+0x4e/0xd0
  470. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bc3b0>] ? bio_alloc_bioset+0x40/0xc0
  471. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b7f94>] submit_bh+0xe4/0x140
  472. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b9fb0>] __block_write_full_page+0x200/0x370
  473. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  474. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01ba1fd>] block_write_full_page_endio+0xdd/0x120
  475. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  476. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bf310>] ? blkdev_get_block+0x0/0x60
  477. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01ba252>] block_write_full_page+0x12/0x20
  478. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01bad40>] ? end_buffer_async_write+0x0/0x160
  479. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01be51f>] blkdev_writepage+0xf/0x20
  480. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0169e1b>] __writepage+0xb/0x30
  481. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a1bd>] write_cache_pages+0x1fd/0x3c0
  482. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0169e10>] ? __writepage+0x0/0x30
  483. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a3a3>] generic_writepages+0x23/0x30
  484. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a380>] ? generic_writepages+0x0/0x30
  485. Jun 13 01:17:14 AGC-A2X2 kernel: [<c016a3cd>] do_writepages+0x1d/0x30
  486. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b29f9>] writeback_single_inode+0x99/0x270
  487. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b347e>] writeback_inodes_wb+0x2ae/0x420
  488. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b36ec>] wb_writeback+0xfc/0x1b0
  489. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b39c8>] wb_do_writeback+0x158/0x180
  490. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01b3a15>] bdi_writeback_task+0x25/0xa0
  491. Jun 13 01:17:14 AGC-A2X2 kernel: [<c017603e>] bdi_start_fn+0x5e/0xd0
  492. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0175fe0>] ? bdi_start_fn+0x0/0xd0
  493. Jun 13 01:17:14 AGC-A2X2 kernel: [<c0142854>] kthread+0x74/0x80
  494. Jun 13 01:17:14 AGC-A2X2 kernel: [<c01427e0>] ? kthread+0x0/0x80
  495. Jun 13 01:17:14 AGC-A2X2 kernel: [<c010483b>] kernel_thread_helper+0x7/0x10
  496. Jun 13 01:17:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  497. Jun 13 01:17:29 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16844 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  498. Jun 13 01:18:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16857 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  499. Jun 13 01:18:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  500. Jun 13 01:18:50 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16875 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  501. Jun 13 01:19:14 AGC-A2X2 kernel: INFO: task ovs-vswitchd:6488 blocked for more than 120 seconds.
  502. Jun 13 01:19:14 AGC-A2X2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  503. Jun 13 01:19:14 AGC-A2X2 kernel: ovs-vswitchd D e4f439c8 0 6488 6487 0x00000004
  504. Jun 13 01:19:14 AGC-A2X2 kernel: e4f439dc 00200286 00000002 e4f439c8 c16eb224 00000000 c04cb780 00000000
  505. Jun 13 01:19:14 AGC-A2X2 kernel: 00000000 e4f439a4 ed7869d4 ed7868c4 ed786830 ed7869d4 c16ec100 00000003
  506. Jun 13 01:19:14 AGC-A2X2 kernel: e614c200 00435dea 00000000 00000000 00000000 e4f439d4 c014a3a6 0f390cfd
  507. Jun 13 01:19:14 AGC-A2X2 kernel: Call Trace:
  508. Jun 13 01:19:14 AGC-A2X2 kernel: [<c014a3a6>] ? ktime_get_ts+0xe6/0x110
  509. Jun 13 01:19:14 AGC-A2X2 kernel: [<c03833fb>] io_schedule+0x4b/0x80
  510. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0163145>] sync_page+0x35/0x40
  511. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0383775>] __wait_on_bit+0x45/0x70
  512. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0163110>] ? sync_page+0x0/0x40
  513. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0163515>] wait_on_page_bit+0x55/0x60
  514. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0142b60>] ? wake_bit_function+0x0/0x50
  515. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016e793>] shrink_page_list+0x463/0x690
  516. Jun 13 01:19:14 AGC-A2X2 kernel: [<c015fdf6>] ? delayacct_end+0x86/0xa0
  517. Jun 13 01:19:14 AGC-A2X2 kernel: [<c015ff6c>] ? __delayacct_blkio_end+0x6c/0x80
  518. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016f0c2>] shrink_list+0x702/0x7e0
  519. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016a54f>] ? determine_dirtyable_memory+0x3f/0x60
  520. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016a585>] ? get_dirty_limits+0x15/0x2d0
  521. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016f534>] shrink_zone+0x274/0x360
  522. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016fa51>] try_to_free_pages+0x1a1/0x2b0
  523. Jun 13 01:19:14 AGC-A2X2 kernel: [<c016d520>] ? isolate_pages_global+0x0/0x1d0
  524. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0169028>] __alloc_pages_nodemask+0x238/0x4f0
  525. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0120e48>] ? __wake_up_common+0x48/0x70
  526. Jun 13 01:19:14 AGC-A2X2 kernel: [<c018f9d2>] cache_alloc_refill+0x312/0x610
  527. Jun 13 01:19:14 AGC-A2X2 kernel: [<c018fe23>] __kmalloc+0x153/0x180
  528. Jun 13 01:19:14 AGC-A2X2 kernel: [<c030014f>] __alloc_skb+0x6f/0x170
  529. Jun 13 01:19:14 AGC-A2X2 kernel: [<c020257f>] ? security_sock_rcv_skb+0xf/0x20
  530. Jun 13 01:19:14 AGC-A2X2 kernel: [<c02fb1cd>] ? sock_alloc_send_pskb+0x15d/0x280
  531. Jun 13 01:19:14 AGC-A2X2 kernel: [<c02fb1cd>] sock_alloc_send_pskb+0x15d/0x280
  532. Jun 13 01:19:14 AGC-A2X2 kernel: [<c02fb308>] sock_alloc_send_skb+0x18/0x20
  533. Jun 13 01:19:14 AGC-A2X2 kernel: [<c036ccd4>] unix_stream_sendmsg+0x1c4/0x350
  534. Jun 13 01:19:14 AGC-A2X2 kernel: [<c032ac12>] ? ip_rcv+0x1e2/0x2c0
  535. Jun 13 01:19:14 AGC-A2X2 kernel: [<c02f7f3a>] sock_aio_write+0x10a/0x120
  536. Jun 13 01:19:14 AGC-A2X2 kernel: [<c019631d>] do_sync_write+0xcd/0x110
  537. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0142b10>] ? autoremove_wake_function+0x0/0x50
  538. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0201aaf>] ? security_file_permission+0xf/0x20
  539. Jun 13 01:19:14 AGC-A2X2 kernel: [<c01964d0>] ? rw_verify_area+0x60/0xc0
  540. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0196b56>] vfs_write+0x146/0x150
  541. Jun 13 01:19:14 AGC-A2X2 kernel: [<c02f9bfc>] ? sys_socketcall+0x25c/0x270
  542. Jun 13 01:19:14 AGC-A2X2 kernel: [<c01970fd>] sys_write+0x3d/0x70
  543. Jun 13 01:19:14 AGC-A2X2 kernel: [<c0104501>] syscall_call+0x7/0xb
  544. Jun 13 01:19:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  545. Jun 13 01:19:30 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16888 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  546. Jun 13 01:20:10 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 16901 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  547. Jun 13 01:20:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  548. Jun 13 01:21:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  549. Jun 13 01:22:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  550. Jun 13 01:23:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  551. Jun 13 01:24:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  552. Jun 13 01:25:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  553. Jun 13 01:26:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  554. Jun 13 01:27:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  555. Jun 13 01:28:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  556. Jun 13 01:29:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  557. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3605920, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  558. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606008, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  559. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606096, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  560. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606184, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  561. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606272, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  562. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606360, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  563. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606448, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  564. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606536, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  565. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606624, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  566. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606712, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  567. Jun 13 01:30:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606800, secs: 48, nbytes: 24576, blk: 880, blk_offset: 3612767
  568. Jun 13 01:30:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  569. Jun 13 01:31:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  570. Jun 13 01:32:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  571. Jun 13 01:33:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  572. Jun 13 01:34:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  573. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: cirrus: blanking the screen line_offset=1024 height=768
  574. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENUTIL Notice 147(0): FFFFF96000810000 - FFFFF96000819FFF: vga.dll
  575. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENUTIL Notice 148(0): FFFFF960009E0000 - FFFFF960009E8FFF: framebuf.dll
  576. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENUTIL Notice 149(0): FFFFF960008C0000 - FFFFF960008C9FFF: vga.dll
  577. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENUTIL Notice 150(0): FFFFF96000940000 - FFFFF96000949FFF: vga.dll
  578. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENUTIL Notice 151(0): FFFFF96000830000 - FFFFF96000838FFF: framebuf.dll
  579. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: cirrus: blanking the screen line_offset=3072 height=768
  580. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Console changed
  581. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : changed to session 2
  582. Jun 13 01:35:04 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Restart worker
  583. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Stopping worker process 1316
  584. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Communications broken, resetting client : No bytes to read, pipe closed
  585. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : New session 2
  586. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Got new session token
  587. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Spawn Worker Process
  588. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Worker Process spawned
  589. Jun 13 01:35:05 AGC-A2X2 fe: qemu-dm-8[22268]: XENIFACE Notice USER: Clipboard : Worker client connected
  590. Jun 13 01:35:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  591. Jun 13 01:36:07 AGC-A2X2 /opt/xensource/bin/xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|http] Failed to parse HTTP response status line []
  592. Jun 13 01:36:07 AGC-A2X2 /opt/xensource/bin/xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Http_client.Http_request_rejected("{ method = POST; uri = /remote_db_access; query = [ ]; content_length = [ 193 ]; transfer encoding = ; version = 1.1; cookie = [ pool_secret=d923de3f-0b51-195e-deb1-3d5e60d168e9/27323cfb-0fa3-fbc2-d6fb-0425fd44ae04/004b3934-87fb-5e75-ec6f-78fca007c5d7 ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  593. Jun 13 01:36:07 AGC-A2X2 /opt/xensource/bin/xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  594. Jun 13 01:36:12 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17221 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  595. Jun 13 01:36:12 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  596. Jun 13 01:36:12 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  597. Jun 13 01:36:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  598. Jun 13 01:36:15 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17229 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  599. Jun 13 01:36:15 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  600. Jun 13 01:36:15 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  601. Jun 13 01:36:27 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17237 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  602. Jun 13 01:36:27 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  603. Jun 13 01:36:27 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  604. Jun 13 01:36:32 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17245 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  605. Jun 13 01:36:32 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  606. Jun 13 01:36:32 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  607. Jun 13 01:36:35 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17253 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  608. Jun 13 01:36:35 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  609. Jun 13 01:36:35 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  610. Jun 13 01:36:40 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17261 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  611. Jun 13 01:36:40 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  612. Jun 13 01:36:40 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  613. Jun 13 01:36:43 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17269 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  614. Jun 13 01:36:43 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  615. Jun 13 01:36:43 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  616. Jun 13 01:36:55 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17277 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  617. Jun 13 01:36:55 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  618. Jun 13 01:36:55 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  619. Jun 13 01:37:00 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17285 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  620. Jun 13 01:37:00 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  621. Jun 13 01:37:00 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  622. Jun 13 01:37:03 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17302 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  623. Jun 13 01:37:03 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  624. Jun 13 01:37:03 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  625. Jun 13 01:37:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3606920, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  626. Jun 13 01:37:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3607008, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  627. Jun 13 01:37:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3607096, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  628. Jun 13 01:37:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3607184, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  629. Jun 13 01:37:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 2, lsec: 3607272, secs: 88, nbytes: 45056, blk: 880, blk_offset: 3612767
  630. Jun 13 01:37:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  631. Jun 13 01:37:15 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17310 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  632. Jun 13 01:37:15 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  633. Jun 13 01:37:15 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  634. Jun 13 01:37:20 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17318 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  635. Jun 13 01:37:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  636. Jun 13 01:37:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  637. Jun 13 01:37:23 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17326 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  638. Jun 13 01:37:23 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  639. Jun 13 01:37:23 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  640. Jun 13 01:37:28 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17334 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  641. Jun 13 01:37:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  642. Jun 13 01:37:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  643. Jun 13 01:37:31 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17342 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  644. Jun 13 01:37:31 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  645. Jun 13 01:37:31 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  646. Jun 13 01:37:36 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17350 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  647. Jun 13 01:37:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  648. Jun 13 01:37:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  649. Jun 13 01:37:39 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17358 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  650. Jun 13 01:37:39 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  651. Jun 13 01:37:39 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  652. Jun 13 01:37:51 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17366 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  653. Jun 13 01:37:51 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  654. Jun 13 01:37:51 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  655. Jun 13 01:37:56 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17374 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  656. Jun 13 01:37:56 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  657. Jun 13 01:37:56 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  658. Jun 13 01:37:59 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17391 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  659. Jun 13 01:37:59 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  660. Jun 13 01:37:59 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  661. Jun 13 01:38:11 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17409 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  662. Jun 13 01:38:11 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  663. Jun 13 01:38:11 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  664. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/773bac2e-c3f7-bfcc-8003-bfda632a67b3/2ac43a0e-5099-4c22-8335-71c523970c64.vhd: op: 4, lsec: 3604480, secs: 1, nbytes: 512, blk: 880, blk_offset: 3612767
  665. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.0: write 0x0058 secs @ 0x003705a0 - Input/output error
  666. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.0 timed out, retried 0 times
  667. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.1: write 0x0058 secs @ 0x003705f8 - Input/output error
  668. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.1 timed out, retried 0 times
  669. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.2: write 0x0058 secs @ 0x00370650 - Input/output error
  670. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.2 timed out, retried 0 times
  671. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.3: write 0x0058 secs @ 0x003706a8 - Input/output error
  672. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.3 timed out, retried 0 times
  673. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.4: write 0x0058 secs @ 0x00370700 - Input/output error
  674. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.4 timed out, retried 0 times
  675. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.5: write 0x0058 secs @ 0x00370758 - Input/output error
  676. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.5 timed out, retried 0 times
  677. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.6: write 0x0058 secs @ 0x003707b0 - Input/output error
  678. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.6 timed out, retried 0 times
  679. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.7: write 0x0058 secs @ 0x00370808 - Input/output error
  680. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.7 timed out, retried 0 times
  681. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.8: write 0x0058 secs @ 0x00370860 - Input/output error
  682. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.8 timed out, retried 0 times
  683. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.9: write 0x0058 secs @ 0x003708b8 - Input/output error
  684. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.9 timed out, retried 0 times
  685. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.10: write 0x0030 secs @ 0x00370910 - Input/output error
  686. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.10 timed out, retried 0 times
  687. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.11: write 0x0058 secs @ 0x00370988 - Input/output error
  688. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.11 timed out, retried 0 times
  689. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.12: write 0x0058 secs @ 0x003709e0 - Input/output error
  690. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.12 timed out, retried 0 times
  691. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.13: write 0x0058 secs @ 0x00370a38 - Input/output error
  692. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.13 timed out, retried 0 times
  693. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.14: write 0x0058 secs @ 0x00370a90 - Input/output error
  694. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.14 timed out, retried 0 times
  695. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_complete_td_request: req tap-7.15: write 0x0058 secs @ 0x00370ae8 - Input/output error
  696. Jun 13 01:38:15 AGC-A2X2 tapdisk[15206]: ERROR: errno -5 at __tapdisk_vbd_request_timeout: req tap-7.15 timed out, retried 0 times
  697. Jun 13 01:38:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  698. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3605920
  699. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3605920
  700. Jun 13 01:38:15 AGC-A2X2 kernel: __ratelimit: 6 callbacks suppressed
  701. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450740
  702. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  703. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450741
  704. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  705. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450742
  706. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  707. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450743
  708. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  709. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450744
  710. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  711. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450745
  712. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  713. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450746
  714. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  715. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450747
  716. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  717. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450748
  718. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  719. Jun 13 01:38:15 AGC-A2X2 kernel: Buffer I/O error on device xvda, logical block 450749
  720. Jun 13 01:38:15 AGC-A2X2 kernel: lost page write due to I/O error on xvda
  721. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606008
  722. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606008
  723. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606096
  724. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606096
  725. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606184
  726. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606184
  727. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606272
  728. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606272
  729. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606360
  730. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606360
  731. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606448
  732. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606448
  733. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606536
  734. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606536
  735. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606624
  736. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606624
  737. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606712
  738. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606712
  739. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606800
  740. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606800
  741. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3606920
  742. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3606920
  743. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3607008
  744. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3607008
  745. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3607096
  746. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3607096
  747. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3607184
  748. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3607184
  749. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev tdh, sector 3607272
  750. Jun 13 01:38:15 AGC-A2X2 kernel: end_request: I/O error, dev xvda, sector 3607272
  751. Jun 13 01:38:16 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17421 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  752. Jun 13 01:38:16 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  753. Jun 13 01:38:16 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  754. Jun 13 01:38:19 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17429 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  755. Jun 13 01:38:19 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  756. Jun 13 01:38:19 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  757. Jun 13 01:38:25 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17437 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  758. Jun 13 01:38:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  759. Jun 13 01:38:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  760. Jun 13 01:38:28 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17445 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  761. Jun 13 01:38:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  762. Jun 13 01:38:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  763. Jun 13 01:38:33 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17453 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  764. Jun 13 01:38:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  765. Jun 13 01:38:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  766. Jun 13 01:38:36 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17461 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  767. Jun 13 01:38:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  768. Jun 13 01:38:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  769. Jun 13 01:38:41 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17469 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  770. Jun 13 01:38:41 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  771. Jun 13 01:38:41 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  772. Jun 13 01:38:44 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17477 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  773. Jun 13 01:38:44 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  774. Jun 13 01:38:44 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  775. Jun 13 01:38:56 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17485 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  776. Jun 13 01:38:56 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  777. Jun 13 01:38:56 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  778. Jun 13 01:39:01 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17502 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  779. Jun 13 01:39:01 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  780. Jun 13 01:39:01 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  781. Jun 13 01:39:04 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17510 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  782. Jun 13 01:39:04 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  783. Jun 13 01:39:04 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  784. Jun 13 01:39:09 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17518 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  785. Jun 13 01:39:09 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  786. Jun 13 01:39:09 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  787. Jun 13 01:39:12 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17526 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  788. Jun 13 01:39:12 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  789. Jun 13 01:39:12 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  790. Jun 13 01:39:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  791. Jun 13 01:39:17 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17534 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  792. Jun 13 01:39:17 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  793. Jun 13 01:39:17 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  794. Jun 13 01:39:20 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17542 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  795. Jun 13 01:39:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  796. Jun 13 01:39:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  797. Jun 13 01:39:25 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17550 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  798. Jun 13 01:39:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  799. Jun 13 01:39:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  800. Jun 13 01:39:28 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17558 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  801. Jun 13 01:39:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  802. Jun 13 01:39:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  803. Jun 13 01:39:33 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17566 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  804. Jun 13 01:39:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  805. Jun 13 01:39:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  806. Jun 13 01:39:36 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17574 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  807. Jun 13 01:39:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  808. Jun 13 01:39:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  809. Jun 13 01:39:41 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17582 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  810. Jun 13 01:39:41 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  811. Jun 13 01:39:41 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  812. Jun 13 01:39:44 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17590 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  813. Jun 13 01:39:44 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  814. Jun 13 01:39:44 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  815. Jun 13 01:39:49 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17598 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  816. Jun 13 01:39:49 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  817. Jun 13 01:39:49 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  818. Jun 13 01:39:52 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17606 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  819. Jun 13 01:39:52 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  820. Jun 13 01:39:52 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  821. Jun 13 01:39:57 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17614 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  822. Jun 13 01:39:57 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  823. Jun 13 01:39:57 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  824. Jun 13 01:40:00 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17632 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  825. Jun 13 01:40:00 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  826. Jun 13 01:40:00 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  827. Jun 13 01:40:05 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17662 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  828. Jun 13 01:40:05 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  829. Jun 13 01:40:05 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  830. Jun 13 01:40:08 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17674 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  831. Jun 13 01:40:08 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  832. Jun 13 01:40:08 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  833. Jun 13 01:40:15 AGC-A2X2 kernel: nfs: server 10.0.50.50 not responding, timed out
  834. Jun 13 01:40:20 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17682 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  835. Jun 13 01:40:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  836. Jun 13 01:40:20 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  837. Jun 13 01:40:25 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17691 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  838. Jun 13 01:40:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  839. Jun 13 01:40:25 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  840. Jun 13 01:40:28 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17699 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  841. Jun 13 01:40:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  842. Jun 13 01:40:28 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  843. Jun 13 01:40:33 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17708 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  844. Jun 13 01:40:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  845. Jun 13 01:40:33 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
  846. Jun 13 01:40:36 AGC-A2X2 xapi: [ warn|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|xapi] stunnel pid: 17716 caught Http_client.Http_request_rejected("{ method = POST; uri = /; query = [ ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; user_agent = xapi/1.9 }")
  847. Jun 13 01:40:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Caught Unix.Unix_error(31, "write", "")
  848. Jun 13 01:40:36 AGC-A2X2 xapi: [error|AGC-A2X2|25 heartbeat|Heartbeat D:0d759973ae08|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!