API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jun 8th, 2013
28
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 29.42 KB | None | 0 0
raw download clone embed print report
  1. root@ceph-osd2:~# ceph -w
  2. health HEALTH_ERR 74 pgs incomplete; 182 pgs inconsistent; 74 pgs stuck inactive; 101 pgs stuck unclean; 326 scrub errors; mds cluster is degraded; mds a is laggy
  3. monmap e13: 3 mons at {b=10.11.3.3:6789/0,c=10.11.3.4:6789/0,d=10.11.3.6:6789/0}, election epoch 3902, quorum 0,1,2 b,c,d
  4. osdmap e10327: 21 osds: 19 up, 19 in
  5. pgmap v905012: 1400 pgs: 1126 active+clean, 5 inconsistent+incomplete, 23 active+remapped, 68 incomplete, 173 active+clean+inconsistent, 1 remapped+incomplete, 4 active+remapped+inconsistent; 2575 GB data, 7597 GB used, 19402 GB / 27000 GB avail
  6. mdsmap e318023: 1/1/1 up {0=a=up:replay(laggy or crashed)}
  7.  
  8. 2013-06-08 21:08:00.040860 osd.20 [WRN] slow request 4647.153498 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  9. 2013-06-08 21:08:01.041046 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4651.437813 secs
  10. 2013-06-08 21:08:01.041052 osd.20 [WRN] slow request 4651.437813 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  11. 2013-06-08 21:08:01.041056 osd.20 [WRN] slow request 4650.891133 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  12. 2013-06-08 21:08:01.041059 osd.20 [WRN] slow request 4648.694617 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  13. 2013-06-08 21:08:01.041062 osd.20 [WRN] slow request 4648.537557 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  14. 2013-06-08 21:08:01.041065 osd.20 [WRN] slow request 4648.153704 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  15. 2013-06-08 21:08:02.041251 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4652.438018 secs
  16. 2013-06-08 21:08:02.041257 osd.20 [WRN] slow request 4652.438018 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  17. 2013-06-08 21:08:02.041261 osd.20 [WRN] slow request 4651.891338 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  18. 2013-06-08 21:08:02.041264 osd.20 [WRN] slow request 4649.694822 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  19. 2013-06-08 21:08:02.041269 osd.20 [WRN] slow request 4649.537762 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  20. 2013-06-08 21:08:02.041272 osd.20 [WRN] slow request 4649.153909 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  21. 2013-06-08 21:08:03.041368 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4653.438171 secs
  22. 2013-06-08 21:08:03.041371 osd.20 [WRN] slow request 4653.438171 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  23. 2013-06-08 21:08:03.041373 osd.20 [WRN] slow request 4652.891491 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  24. 2013-06-08 21:08:03.041375 osd.20 [WRN] slow request 4650.694975 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  25. 2013-06-08 21:08:03.041376 osd.20 [WRN] slow request 4650.537915 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  26. 2013-06-08 21:08:03.041378 osd.20 [WRN] slow request 4650.154062 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  27. 2013-06-08 21:08:04.041505 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4654.438308 secs
  28. 2013-06-08 21:08:04.041508 osd.20 [WRN] slow request 4654.438308 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  29. 2013-06-08 21:08:04.041509 osd.20 [WRN] slow request 4653.891628 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  30. 2013-06-08 21:08:04.041511 osd.20 [WRN] slow request 4651.695112 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  31. 2013-06-08 21:08:04.041512 osd.20 [WRN] slow request 4651.538052 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  32. 2013-06-08 21:08:04.041514 osd.20 [WRN] slow request 4651.154199 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  33. 2013-06-08 21:08:05.041674 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4655.438436 secs
  34. 2013-06-08 21:08:05.041680 osd.20 [WRN] slow request 4655.438436 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  35. 2013-06-08 21:08:05.041684 osd.20 [WRN] slow request 4654.891756 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  36. 2013-06-08 21:08:05.041687 osd.20 [WRN] slow request 4652.695240 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  37. 2013-06-08 21:08:05.041691 osd.20 [WRN] slow request 4652.538180 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  38. 2013-06-08 21:08:05.041694 osd.20 [WRN] slow request 4652.154327 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  39. 2013-06-08 21:08:06.041883 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4656.438650 secs
  40. 2013-06-08 21:08:06.041888 osd.20 [WRN] slow request 4656.438650 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  41. 2013-06-08 21:08:06.041892 osd.20 [WRN] slow request 4655.891970 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  42. 2013-06-08 21:08:06.041895 osd.20 [WRN] slow request 4653.695454 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  43. 2013-06-08 21:08:06.041899 osd.20 [WRN] slow request 4653.538394 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  44. 2013-06-08 21:08:06.041902 osd.20 [WRN] slow request 4653.154541 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  45. 2013-06-08 21:08:07.042119 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4657.438889 secs
  46. 2013-06-08 21:08:07.042124 osd.20 [WRN] slow request 4657.438889 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  47. 2013-06-08 21:08:07.042128 osd.20 [WRN] slow request 4656.892209 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  48. 2013-06-08 21:08:07.042132 osd.20 [WRN] slow request 4654.695693 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  49. 2013-06-08 21:08:07.042135 osd.20 [WRN] slow request 4654.538633 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  50. 2013-06-08 21:08:07.042139 osd.20 [WRN] slow request 4654.154780 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  51. 2013-06-08 21:08:08.042317 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4658.439084 secs
  52. 2013-06-08 21:08:08.042322 osd.20 [WRN] slow request 4658.439084 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  53. 2013-06-08 21:08:08.042326 osd.20 [WRN] slow request 4657.892404 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  54. 2013-06-08 21:08:08.042329 osd.20 [WRN] slow request 4655.695888 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  55. 2013-06-08 21:08:08.042332 osd.20 [WRN] slow request 4655.538828 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  56. 2013-06-08 21:08:08.042336 osd.20 [WRN] slow request 4655.154975 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  57. 2013-06-08 21:08:09.042524 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4659.439283 secs
  58. 2013-06-08 21:08:09.042533 osd.20 [WRN] slow request 4659.439283 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  59. 2013-06-08 21:08:09.042540 osd.20 [WRN] slow request 4658.892603 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  60. 2013-06-08 21:08:09.042546 osd.20 [WRN] slow request 4656.696087 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  61. 2013-06-08 21:08:09.042552 osd.20 [WRN] slow request 4656.539027 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  62. 2013-06-08 21:08:09.042558 osd.20 [WRN] slow request 4656.155174 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  63. 2013-06-08 21:08:10.042744 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4660.439511 secs
  64. 2013-06-08 21:08:10.042750 osd.20 [WRN] slow request 4660.439511 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  65. 2013-06-08 21:08:10.042753 osd.20 [WRN] slow request 4659.892831 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  66. 2013-06-08 21:08:10.042757 osd.20 [WRN] slow request 4657.696315 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  67. 2013-06-08 21:08:10.042760 osd.20 [WRN] slow request 4657.539255 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  68. 2013-06-08 21:08:10.042764 osd.20 [WRN] slow request 4657.155402 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  69. 2013-06-08 21:08:18.336868 mon.0 [INF] mdsmap e318024: 1/1/1 up {0=a=up:replay(laggy or crashed)}
  70. 2013-06-08 21:08:11.042956 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4661.439721 secs
  71. 2013-06-08 21:08:11.042961 osd.20 [WRN] slow request 4661.439721 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  72. 2013-06-08 21:08:11.042965 osd.20 [WRN] slow request 4660.893041 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  73. 2013-06-08 21:08:11.042969 osd.20 [WRN] slow request 4658.696525 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  74. 2013-06-08 21:08:11.042972 osd.20 [WRN] slow request 4658.539465 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  75. 2013-06-08 21:08:11.042975 osd.20 [WRN] slow request 4658.155612 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  76. 2013-06-08 21:08:12.043193 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4662.439962 secs
  77. 2013-06-08 21:08:12.043198 osd.20 [WRN] slow request 4662.439962 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  78. 2013-06-08 21:08:12.043202 osd.20 [WRN] slow request 4661.893282 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  79. 2013-06-08 21:08:12.043205 osd.20 [WRN] slow request 4659.696766 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  80. 2013-06-08 21:08:12.043208 osd.20 [WRN] slow request 4659.539706 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  81. 2013-06-08 21:08:12.043233 osd.20 [WRN] slow request 4659.155853 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  82. 2013-06-08 21:08:13.043391 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4663.440158 secs
  83. 2013-06-08 21:08:13.043396 osd.20 [WRN] slow request 4663.440158 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  84. 2013-06-08 21:08:13.043400 osd.20 [WRN] slow request 4662.893478 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  85. 2013-06-08 21:08:13.043403 osd.20 [WRN] slow request 4660.696962 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  86. 2013-06-08 21:08:13.043406 osd.20 [WRN] slow request 4660.539902 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  87. 2013-06-08 21:08:13.043409 osd.20 [WRN] slow request 4660.156049 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  88. 2013-06-08 21:08:14.043622 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4664.440392 secs
  89. 2013-06-08 21:08:14.043628 osd.20 [WRN] slow request 4664.440392 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  90. 2013-06-08 21:08:14.043631 osd.20 [WRN] slow request 4663.893712 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  91. 2013-06-08 21:08:14.043634 osd.20 [WRN] slow request 4661.697196 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  92. 2013-06-08 21:08:14.043638 osd.20 [WRN] slow request 4661.540136 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  93. 2013-06-08 21:08:14.043641 osd.20 [WRN] slow request 4661.156283 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  94. 2013-06-08 21:08:15.043852 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4665.440622 secs
  95. 2013-06-08 21:08:15.043857 osd.20 [WRN] slow request 4665.440622 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  96. 2013-06-08 21:08:15.043861 osd.20 [WRN] slow request 4664.893942 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  97. 2013-06-08 21:08:15.043864 osd.20 [WRN] slow request 4662.697426 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  98. 2013-06-08 21:08:15.043868 osd.20 [WRN] slow request 4662.540366 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  99. 2013-06-08 21:08:15.043871 osd.20 [WRN] slow request 4662.156513 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  100. 2013-06-08 21:08:16.044057 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4666.440824 secs
  101. 2013-06-08 21:08:16.044062 osd.20 [WRN] slow request 4666.440824 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  102. 2013-06-08 21:08:16.044066 osd.20 [WRN] slow request 4665.894144 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  103. 2013-06-08 21:08:16.044070 osd.20 [WRN] slow request 4663.697628 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  104. 2013-06-08 21:08:16.044073 osd.20 [WRN] slow request 4663.540568 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  105. 2013-06-08 21:08:16.044076 osd.20 [WRN] slow request 4663.156715 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  106. 2013-06-08 21:08:17.044264 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4667.441032 secs
  107. 2013-06-08 21:08:17.044269 osd.20 [WRN] slow request 4667.441032 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  108. 2013-06-08 21:08:17.044273 osd.20 [WRN] slow request 4666.894352 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  109. 2013-06-08 21:08:17.044276 osd.20 [WRN] slow request 4664.697836 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  110. 2013-06-08 21:08:17.044280 osd.20 [WRN] slow request 4664.540776 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  111. 2013-06-08 21:08:17.044284 osd.20 [WRN] slow request 4664.156923 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  112. 2013-06-08 21:08:18.044448 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4668.441214 secs
  113. 2013-06-08 21:08:18.044454 osd.20 [WRN] slow request 4668.441214 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  114. 2013-06-08 21:08:18.044458 osd.20 [WRN] slow request 4667.894534 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  115. 2013-06-08 21:08:18.044461 osd.20 [WRN] slow request 4665.698018 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  116. 2013-06-08 21:08:18.044467 osd.20 [WRN] slow request 4665.540958 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  117. 2013-06-08 21:08:18.044509 osd.20 [WRN] slow request 4665.157105 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  118. 2013-06-08 21:08:19.044681 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4669.441439 secs
  119. 2013-06-08 21:08:19.044689 osd.20 [WRN] slow request 4669.441439 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  120. 2013-06-08 21:08:19.044697 osd.20 [WRN] slow request 4668.894759 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  121. 2013-06-08 21:08:19.044703 osd.20 [WRN] slow request 4666.698243 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  122. 2013-06-08 21:08:19.044709 osd.20 [WRN] slow request 4666.541183 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  123. 2013-06-08 21:08:19.044715 osd.20 [WRN] slow request 4666.157330 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  124. 2013-06-08 21:08:20.044897 osd.20 [WRN] 5 slow requests, 5 included below; oldest blocked for > 4670.441656 secs
  125. 2013-06-08 21:08:20.044906 osd.20 [WRN] slow request 4670.441656 seconds old, received at 2013-06-08 19:50:29.603159: osd_op(client.2470731.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  126. 2013-06-08 21:08:20.044913 osd.20 [WRN] slow request 4669.894976 seconds old, received at 2013-06-08 19:50:30.149839: osd_op(client.2464662.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10326) v4 currently reached pg
  127. 2013-06-08 21:08:20.044919 osd.20 [WRN] slow request 4667.698460 seconds old, received at 2013-06-08 19:50:32.346355: osd_op(client.2448680.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  128. 2013-06-08 21:08:20.044925 osd.20 [WRN] slow request 4667.541400 seconds old, received at 2013-06-08 19:50:32.503415: osd_op(client.2448665.0:18 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=5 e10327) v4 currently reached pg
  129. 2013-06-08 21:08:20.044931 osd.20 [WRN] slow request 4667.157547 seconds old, received at 2013-06-08 19:50:32.887268: osd_op(client.2470752.0:4 rbd_header.15646b8b4567 [call rbd.get_size,call rbd.get_object_prefix] 3.777cb787 RETRY=2 e10327) v4 currently reached pg
  130. ^C
  131. root@ceph-osd2:~#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!