API tools faq
paste
Guest User

DMESG while IO WAIT

a guest
Apr 21st, 2015
462
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 29.03 KB | None | 0 0
raw download clone embed print report
  1. Apr 19 08:28:51 ceph-rsync-bs02 kernel: [240412.816143] libceph: osd603 10.76.28.81:6814 socket closed (con state OPEN)
  2. Apr 19 08:47:26 ceph-rsync-bs02 kernel: [241529.129354] libceph: osd601 10.76.28.58:6838 socket closed (con state OPEN)
  3. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792429] INFO: task kswapd0:79 blocked for more than 120 seconds.
  4. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792550] Not tainted 3.16-0.bpo.2-amd64 #1
  5. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  6. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792748] kswapd0 D ffff88062f0341c0 0 79 2 0x00000000
  7. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792754] ffff880610898110 0000000000000046 0000000000000000 ffff8806154a2be0
  8. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792758] 00000000000141c0 ffff8806126d3fd8 00000000000141c0 ffff8806154a2be0
  9. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792762] ffff8806126d3838 ffff88009f071890 7fffffffffffffff ffff88009f071888
  10. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792766] Call Trace:
  11. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792777] [<ffffffff81541f5d>] ? schedule_timeout+0x1dd/0x240
  12. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792783] [<ffffffff8129c3a0>] ? blk_finish_plug+0x10/0x40
  13. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792803] [<ffffffffa04a10b7>] ? _xfs_buf_ioapply+0x277/0x2e0 [xfs]
  14. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792808] [<ffffffff81543d3c>] ? wait_for_completion+0xac/0x120
  15. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792813] [<ffffffff8109e320>] ? try_to_wake_up+0x310/0x310
  16. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792825] [<ffffffffa04a1778>] ? xfs_bwrite+0x28/0x60 [xfs]
  17. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792837] [<ffffffffa04a16f6>] ? xfs_buf_iowait+0x96/0xf0 [xfs]
  18. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792848] [<ffffffffa04a1778>] ? xfs_bwrite+0x28/0x60 [xfs]
  19. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792861] [<ffffffffa04a9224>] ? xfs_reclaim_inode+0x2f4/0x310 [xfs]
  20. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792873] [<ffffffffa04a9437>] ? xfs_reclaim_inodes_ag+0x1f7/0x320 [xfs]
  21. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792886] [<ffffffffa04aa0f1>] ? xfs_reclaim_inodes_nr+0x31/0x40 [xfs]
  22. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792891] [<ffffffff811bdc57>] ? super_cache_scan+0x167/0x170
  23. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792897] [<ffffffff8115b996>] ? shrink_slab_node+0x136/0x2d0
  24. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792901] [<ffffffff811b36ce>] ? vmpressure+0x5e/0xb0
  25. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792906] [<ffffffff8115d8ca>] ? shrink_slab+0x8a/0x130
  26. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792910] [<ffffffff8116131b>] ? balance_pgdat+0x3db/0x5d0
  27. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792915] [<ffffffff8101e49d>] ? native_sched_clock+0x2d/0x80
  28. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792920] [<ffffffff81161677>] ? kswapd+0x167/0x460
  29. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792924] [<ffffffff810af070>] ? __wake_up_sync+0x20/0x20
  30. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792928] [<ffffffff81161510>] ? balance_pgdat+0x5d0/0x5d0
  31. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792933] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  32. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792938] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  33. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792941] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  34. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792945] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  35. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.792963] INFO: task xfsaild/rbd3:13056 blocked for more than 120 seconds.
  36. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793064] Not tainted 3.16-0.bpo.2-amd64 #1
  37. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  38. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793262] xfsaild/rbd3 D ffff88062f0741c0 0 13056 2 0x00000000
  39. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793265] ffff88061669f3b0 0000000000000046 0000000000000010 ffff880610510210
  40. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793269] 00000000000141c0 ffff880611e6bfd8 00000000000141c0 ffff880610510210
  41. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793272] ffffea00104ec701 ffff88062f074a70 ffff880610510210 ffff88061069a7f8
  42. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793276] Call Trace:
  43. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793280] [<ffffffff81542cbc>] ? io_schedule+0x8c/0xd0
  44. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793285] [<ffffffff8129a3cd>] ? get_request+0x3fd/0x7a0
  45. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793289] [<ffffffff812a0ac3>] ? ll_back_merge_fn+0xf3/0x1f0
  46. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793292] [<ffffffff810af070>] ? __wake_up_sync+0x20/0x20
  47. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793297] [<ffffffff8129c49f>] ? blk_queue_bio+0xcf/0x380
  48. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793301] [<ffffffff81297960>] ? generic_make_request+0xb0/0xf0
  49. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793304] [<ffffffff81297a21>] ? submit_bio+0x81/0x160
  50. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793316] [<ffffffffa04a1074>] ? _xfs_buf_ioapply+0x234/0x2e0 [xfs]
  51. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793320] [<ffffffff8109e320>] ? try_to_wake_up+0x310/0x310
  52. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793331] [<ffffffffa04a13ad>] ? xfs_bdstrat_cb+0x7d/0x120 [xfs]
  53. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793342] [<ffffffffa04a12a6>] ? xfs_buf_iorequest+0x56/0xe0 [xfs]
  54. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793353] [<ffffffffa04a13ad>] ? xfs_bdstrat_cb+0x7d/0x120 [xfs]
  55. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793364] [<ffffffffa04a15cb>] ? __xfs_buf_delwri_submit+0x17b/0x210 [xfs]
  56. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793368] [<ffffffff810b37d2>] ? up+0x12/0x50
  57. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793380] [<ffffffffa04a2210>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
  58. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793391] [<ffffffffa04a0476>] ? xfs_buf_trylock+0x36/0xc0 [xfs]
  59. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793402] [<ffffffffa04a2210>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
  60. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793418] [<ffffffffa05066df>] ? xfsaild+0x21f/0x6e0 [xfs]
  61. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793432] [<ffffffffa05064c0>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
  62. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793436] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  63. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793440] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  64. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793444] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  65. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793448] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  66. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793453] INFO: task rsync:29392 blocked for more than 120 seconds.
  67. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793553] Not tainted 3.16-0.bpo.2-amd64 #1
  68. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793648] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  69. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793750] rsync D ffff88062f0141c0 0 29392 29228 0x00000000
  70. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793753] ffffffff81819480 0000000000000086 0000000000000020 ffff8806104a2050
  71. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793757] 00000000000141c0 ffff8800c8c8bfd8 00000000000141c0 ffff8806104a2050
  72. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793760] 0000000000000000 ffff8804c1a5c6f0 7fffffffffffffff 7fffffffffffffff
  73. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793764] Call Trace:
  74. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793768] [<ffffffff81541f5d>] ? schedule_timeout+0x1dd/0x240
  75. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793785] [<ffffffffa04da94e>] ? xfs_da_read_buf+0xde/0x110 [xfs]
  76. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793790] [<ffffffff81544b61>] ? __down_common+0x97/0xea
  77. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793802] [<ffffffffa04a06ea>] ? _xfs_buf_find+0xea/0x280 [xfs]
  78. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793805] [<ffffffff810b37ac>] ? down+0x3c/0x50
  79. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793816] [<ffffffffa04a0537>] ? xfs_buf_lock+0x37/0x100 [xfs]
  80. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793827] [<ffffffffa04a06ea>] ? _xfs_buf_find+0xea/0x280 [xfs]
  81. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793838] [<ffffffffa04a095a>] ? xfs_buf_get_map+0x3a/0x1a0 [xfs]
  82. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793850] [<ffffffffa04a18f8>] ? xfs_buf_read_map+0x38/0x140 [xfs]
  83. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793864] [<ffffffffa0507c6a>] ? xfs_trans_read_buf_map+0x34a/0x500 [xfs]
  84. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793879] [<ffffffffa04f47ee>] ? xfs_imap_to_bp+0x6e/0xf0 [xfs]
  85. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793895] [<ffffffffa04ed220>] ? xfs_iunlink_remove+0x290/0x390 [xfs]
  86. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793909] [<ffffffffa04fee40>] ? xlog_grant_head_check+0x50/0x100 [xfs]
  87. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793914] [<ffffffff811a0850>] ? kmem_cache_alloc+0x1d0/0x1e0
  88. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793928] [<ffffffffa04f05e6>] ? xfs_ifree+0x26/0xd0 [xfs]
  89. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793943] [<ffffffffa04b8f5a>] ? xfs_trans_add_item+0x2a/0x70 [xfs]
  90. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793956] [<ffffffffa04f074c>] ? xfs_inactive_ifree+0xbc/0x230 [xfs]
  91. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793969] [<ffffffffa04f0971>] ? xfs_inactive+0xb1/0x190 [xfs]
  92. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793975] [<ffffffff811d496c>] ? evict+0xac/0x170
  93. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793980] [<ffffffff811d07c8>] ? __dentry_kill+0x1b8/0x220
  94. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793984] [<ffffffff811d0d4e>] ? dput+0xce/0x1a0
  95. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793989] [<ffffffff811c8680>] ? SYSC_renameat2+0x3d0/0x460
  96. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793993] [<ffffffff811bc6bb>] ? __sb_end_write+0x2b/0x70
  97. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.793997] [<ffffffff811b7d0b>] ? chmod_common+0xbb/0x120
  98. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794001] [<ffffffff811d0cb1>] ? dput+0x31/0x1a0
  99. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794005] [<ffffffff8154646d>] ? system_call_fast_compare_end+0x10/0x15
  100. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794012] INFO: task rsync:23200 blocked for more than 120 seconds.
  101. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794111] Not tainted 3.16-0.bpo.2-amd64 #1
  102. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  103. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794308] rsync D ffff88062f0b41c0 0 23200 23192 0x00000000
  104. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794311] ffff88061669e110 0000000000000086 ffff88051dfe2020 ffff880614eab2f0
  105. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794315] 00000000000141c0 ffff8800a2507fd8 00000000000141c0 ffff880614eab2f0
  106. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794318] ffff8801067c8b38 ffff880277d18e50 ffff880277d18e54 ffff880614eab2f0
  107. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794321] Call Trace:
  108. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794326] [<ffffffff81542eae>] ? schedule_preempt_disabled+0xe/0x20
  109. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794331] [<ffffffff815449b2>] ? __mutex_lock_slowpath+0xd2/0x1c0
  110. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794336] [<ffffffff81544abb>] ? mutex_lock+0x1b/0x2a
  111. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794341] [<ffffffff811c4573>] ? lock_rename+0xd3/0xe0
  112. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794345] [<ffffffff811c844f>] ? SYSC_renameat2+0x19f/0x460
  113. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794349] [<ffffffff811bc6bb>] ? __sb_end_write+0x2b/0x70
  114. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794353] [<ffffffff811b7d0b>] ? chmod_common+0xbb/0x120
  115. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794357] [<ffffffff811d0cb1>] ? dput+0x31/0x1a0
  116. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794360] [<ffffffff8154646d>] ? system_call_fast_compare_end+0x10/0x15
  117. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794363] INFO: task kworker/2:0:18965 blocked for more than 120 seconds.
  118. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794463] Not tainted 3.16-0.bpo.2-amd64 #1
  119. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794558] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  120. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794704] kworker/2:0 D ffff88062f0541c0 0 18965 2 0x00000000
  121. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794713] Workqueue: events handle_timeout [libceph]
  122. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794714] ffff8806166940d0 0000000000000046 00000000000141c0 ffff88061301ab20
  123. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794718] 00000000000141c0 ffff880001973fd8 00000000000141c0 ffff88061301ab20
  124. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794721] 0000000200000002 ffff8804f2bcd9b8 ffff8804f2bcd9bc ffff88061301ab20
  125. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794724] Call Trace:
  126. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794729] [<ffffffff81542eae>] ? schedule_preempt_disabled+0xe/0x20
  127. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794734] [<ffffffff815449b2>] ? __mutex_lock_slowpath+0xd2/0x1c0
  128. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794738] [<ffffffff81544abb>] ? mutex_lock+0x1b/0x2a
  129. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794744] [<ffffffffa064a28e>] ? ceph_con_keepalive+0x2e/0xb0 [libceph]
  130. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794750] [<ffffffffa0652fc1>] ? handle_timeout+0x161/0x1e0 [libceph]
  131. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794756] [<ffffffff8108687c>] ? process_one_work+0x15c/0x450
  132. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794761] [<ffffffff81087ad2>] ? worker_thread+0x112/0x540
  133. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794766] [<ffffffff810879c0>] ? create_and_start_worker+0x60/0x60
  134. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794770] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  135. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794774] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  136. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794778] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  137. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794782] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  138. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794785] INFO: task rsync:19260 blocked for more than 120 seconds.
  139. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.794906] Not tainted 3.16-0.bpo.2-amd64 #1
  140. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  141. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795168] rsync D ffff88062f0341c0 0 19260 19233 0x00000000
  142. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795170] ffff880616694a20 0000000000000082 0000000000000000 ffff8806131bb3f0
  143. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795174] 00000000000141c0 ffff8800c863ffd8 00000000000141c0 ffff8806131bb3f0
  144. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795177] 0000000000000000 ffff880277d18e50 ffff880277d18e54 ffff8806131bb3f0
  145. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795180] Call Trace:
  146. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795185] [<ffffffff81542eae>] ? schedule_preempt_disabled+0xe/0x20
  147. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795189] [<ffffffff815449b2>] ? __mutex_lock_slowpath+0xd2/0x1c0
  148. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795194] [<ffffffff81544abb>] ? mutex_lock+0x1b/0x2a
  149. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795197] [<ffffffff811c91ea>] ? do_last+0x2fa/0xef0
  150. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795201] [<ffffffff811c59fb>] ? link_path_walk+0x5b/0x880
  151. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795205] [<ffffffff811c9eb6>] ? path_openat+0xd6/0x630
  152. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795209] [<ffffffff811c83a6>] ? SYSC_renameat2+0xf6/0x460
  153. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795213] [<ffffffff811ca97d>] ? do_filp_open+0x4d/0xc0
  154. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795217] [<ffffffff811d6f95>] ? __alloc_fd+0x45/0x130
  155. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795222] [<ffffffff811b954e>] ? do_sys_open+0x13e/0x240
  156. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795225] [<ffffffff8154646d>] ? system_call_fast_compare_end+0x10/0x15
  157. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795228] INFO: task kworker/u96:0:19373 blocked for more than 120 seconds.
  158. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795372] Not tainted 3.16-0.bpo.2-amd64 #1
  159. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795488] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  160. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795633] kworker/u96:0 D ffff88062f0741c0 0 19373 2 0x00000000
  161. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795640] Workqueue: writeback bdi_writeback_workfn (flush-253:48)
  162. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795642] ffff88061669f3b0 0000000000000046 0000000000000000 ffff88061204e390
  163. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795645] 00000000000141c0 ffff88000a97bfd8 00000000000141c0 ffff88061204e390
  164. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795649] 0000000000000001 ffff88062f074a70 ffff88061204e390 ffff88061069a7f8
  165. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795652] Call Trace:
  166. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795656] [<ffffffff81542cbc>] ? io_schedule+0x8c/0xd0
  167. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795660] [<ffffffff8129a3cd>] ? get_request+0x3fd/0x7a0
  168. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795664] [<ffffffff810af070>] ? __wake_up_sync+0x20/0x20
  169. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795668] [<ffffffff8129c49f>] ? blk_queue_bio+0xcf/0x380
  170. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795672] [<ffffffff81297960>] ? generic_make_request+0xb0/0xf0
  171. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795676] [<ffffffff81297a21>] ? submit_bio+0x81/0x160
  172. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795686] [<ffffffffa049a3bd>] ? xfs_submit_ioend+0xcd/0x140 [xfs]
  173. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795696] [<ffffffffa049a6ce>] ? xfs_vm_writepage+0x29e/0x560 [xfs]
  174. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795701] [<ffffffff81183b80>] ? SyS_msync+0x280/0x280
  175. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795707] [<ffffffff8115529f>] ? __writepage+0xf/0x40
  176. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795712] [<ffffffff811559b7>] ? write_cache_pages+0x1d7/0x420
  177. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795717] [<ffffffff81155290>] ? set_page_dirty_lock+0x60/0x60
  178. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795722] [<ffffffff81155c48>] ? generic_writepages+0x48/0x70
  179. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795726] [<ffffffff811e1b02>] ? __writeback_single_inode+0x42/0x280
  180. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795730] [<ffffffff811e2f41>] ? writeback_sb_inodes+0x1b1/0x3f0
  181. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795734] [<ffffffff811e321e>] ? __writeback_inodes_wb+0x9e/0xd0
  182. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795738] [<ffffffff811e34db>] ? wb_writeback+0x28b/0x340
  183. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795742] [<ffffffff812d1b0c>] ? vsnprintf+0x1bc/0x5a0
  184. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795747] [<ffffffff811d4173>] ? get_nr_inodes+0x43/0x60
  185. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795751] [<ffffffff811e362f>] ? wb_check_old_data_flush+0x9f/0xb0
  186. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795755] [<ffffffff811e5cc2>] ? bdi_writeback_workfn+0x1a2/0x420
  187. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795760] [<ffffffff8108687c>] ? process_one_work+0x15c/0x450
  188. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795765] [<ffffffff81087ad2>] ? worker_thread+0x112/0x540
  189. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795770] [<ffffffff810879c0>] ? create_and_start_worker+0x60/0x60
  190. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795773] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  191. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795778] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  192. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795781] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  193. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795785] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  194. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795788] INFO: task kworker/4:2:19404 blocked for more than 120 seconds.
  195. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.795910] Not tainted 3.16-0.bpo.2-amd64 #1
  196. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796026] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  197. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796171] kworker/4:2 D ffff88062f0941c0 0 19404 2 0x00000000
  198. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796187] Workqueue: xfs-cil/rbd3 xlog_cil_push_work [xfs]
  199. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796189] ffff880615d9b4f0 0000000000000046 ffff8800a0abfb58 ffff880037044960
  200. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796192] 00000000000141c0 ffff8800a0abffd8 00000000000141c0 ffff880037044960
  201. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796195] ffffffff8109e320 ffff8806114c7800 ffff88060f8fbd08 ffff8806114c7900
  202. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796198] Call Trace:
  203. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796198] Call Trace:
  204. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796202] [<ffffffff8109e320>] ? try_to_wake_up+0x310/0x310
  205. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796216] [<ffffffffa050092c>] ? xlog_state_get_iclog_space+0xdc/0x2c0 [xfs]
  206. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796220] [<ffffffff8109e320>] ? try_to_wake_up+0x310/0x310
  207. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796232] [<ffffffffa0500dea>] ? xlog_write+0x18a/0x6e0 [xfs]
  208. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796246] [<ffffffffa0502b06>] ? xlog_cil_push+0x276/0x380 [xfs]
  209. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796251] [<ffffffff8108687c>] ? process_one_work+0x15c/0x450
  210. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796256] [<ffffffff81086b9c>] ? process_scheduled_works+0x2c/0x40
  211. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796260] [<ffffffff81087c14>] ? worker_thread+0x254/0x540
  212. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796265] [<ffffffff810879c0>] ? create_and_start_worker+0x60/0x60
  213. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796269] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  214. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796273] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  215. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796276] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  216. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796280] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  217. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796283] INFO: task kworker/4:0:19456 blocked for more than 120 seconds.
  218. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796416] Not tainted 3.16-0.bpo.2-amd64 #1
  219. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796533] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  220. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796684] kworker/4:0 D ffff88062f0941c0 0 19456 2 0x00000000
  221. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796707] Workqueue: xfs-log/rbd3 xfs_log_worker [xfs]
  222. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796721] ffff880037044960 0000000000000046 ffff8800a0b83ac8 ffff880615f7a9a0
  223. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796753] 00000000000141c0 ffff8800a0b83fd8 00000000000141c0 ffff880615f7a9a0
  224. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796784] 0000000000000009 ffff8800a0b83c60 7fffffffffffffff ffff8800a0b83c58
  225. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796800] Call Trace:
  226. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796809] [<ffffffff81541f5d>] ? schedule_timeout+0x1dd/0x240
  227. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796816] [<ffffffff81543d3c>] ? wait_for_completion+0xac/0x120
  228. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796822] [<ffffffff8109e320>] ? try_to_wake_up+0x310/0x310
  229. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796829] [<ffffffff810874df>] ? flush_work+0x13f/0x1f0
  230. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796836] [<ffffffff810845e0>] ? worker_detach_from_pool+0xd0/0xd0
  231. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796856] [<ffffffffa0503287>] ? xlog_cil_force_lsn+0x77/0x200 [xfs]
  232. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796876] [<ffffffffa05014c2>] ? _xfs_log_force+0x72/0x2c0 [xfs]
  233. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796883] [<ffffffff8101e49d>] ? native_sched_clock+0x2d/0x80
  234. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796888] [<ffffffff8101e4f5>] ? sched_clock+0x5/0x10
  235. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796895] [<ffffffff810135e5>] ? __switch_to+0x105/0x5b0
  236. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796914] [<ffffffffa0501749>] ? xfs_log_force+0x39/0xe0 [xfs]
  237. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796933] [<ffffffffa0501810>] ? xfs_log_worker+0x20/0x50 [xfs]
  238. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796940] [<ffffffff8108687c>] ? process_one_work+0x15c/0x450
  239. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796959] [<ffffffff81087ad2>] ? worker_thread+0x112/0x540
  240. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796977] [<ffffffff810879c0>] ? create_and_start_worker+0x60/0x60
  241. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.796994] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  242. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797012] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  243. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797029] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  244. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797047] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  245. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797054] INFO: task kworker/4:1:19518 blocked for more than 120 seconds.
  246. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797181] Not tainted 3.16-0.bpo.2-amd64 #1
  247. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  248. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797453] kworker/4:1 D ffff88062f0941c0 0 19518 2 0x00000000
  249. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797467] Workqueue: events handle_osds_timeout [libceph]
  250. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797470] ffff88061669ea60 0000000000000046 ffff88062ec03040 ffff880612214210
  251. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797475] 00000000000141c0 ffff8800a1b2ffd8 00000000000141c0 ffff880612214210
  252. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797480] 000000000000000a ffff8800c8e327c8 ffff8800c8e327cc ffff880612214210
  253. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797492] Call Trace:
  254. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797492] Call Trace:
  255. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797501] [<ffffffff81542eae>] ? schedule_preempt_disabled+0xe/0x20
  256. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797512] [<ffffffff815449b2>] ? __mutex_lock_slowpath+0xd2/0x1c0
  257. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797522] [<ffffffff81544abb>] ? mutex_lock+0x1b/0x2a
  258. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797531] [<ffffffffa0651ebb>] ? handle_osds_timeout+0x7b/0x160 [libceph]
  259. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797539] [<ffffffff8108687c>] ? process_one_work+0x15c/0x450
  260. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797546] [<ffffffff81087ad2>] ? worker_thread+0x112/0x540
  261. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797552] [<ffffffff810879c0>] ? create_and_start_worker+0x60/0x60
  262. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797559] [<ffffffff8108e481>] ? kthread+0xc1/0xe0
  263. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797565] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  264. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797571] [<ffffffff815463bc>] ? ret_from_fork+0x7c/0xb0
  265. Apr 19 09:01:53 ceph-rsync-bs02 kernel: [242396.797577] [<ffffffff8108e3c0>] ? flush_kthread_worker+0xb0/0xb0
  266. Apr 19 09:10:21 ceph-rsync-bs02 kernel: [242905.432273] libceph: osd601 10.76.28.58:6838 socket closed (con state OPEN)
  267. Apr 19 09:10:45 ceph-rsync-bs02 kernel: [242928.879126] libceph: osd710 10.76.28.58:6846 socket closed (con state OPEN)
  268. Apr 19 09:11:13 ceph-rsync-bs02 kernel: [242956.895301] libceph: osd628 10.76.28.58:6802 socket closed (con state OPEN)
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!