API tools faq
paste
Advertisement
Guest User

during last configuration test (RAM DISK)

a guest
May 2nd, 2014
250
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 28.91 KB | None | 0 0
raw download clone embed print report
  1. [root@ds-07-01 ~]# ceph -w
  2. cluster 1f0041f1-93d8-4da7-a859-8d7ae0531e4c
  3. health HEALTH_WARN 6 pgs recovering; 13 pgs recovery_wait; 597 requests are blocked > 32 sec; recovery 91/4348880 objects degraded (0.002%)
  4. monmap e5: 3 mons at {ds-07-01=131.154.129.182:6789/0,ds-07-02=131.154.129.183:6789/0,ds-07-03=131.154.129.184:6789/0}, election epoch 20, quorum 0,1,2 ds-07-01,ds-07-02,ds-07-03
  5. mdsmap e42728: 1/1/1 up {0=ds-07-05=up:active}
  6. osdmap e242: 10 osds: 10 up, 10 in
  7. pgmap v71867: 1024 pgs, 3 pools, 8493 GB data, 2123 kobjects
  8. 17009 GB used, 56342 GB / 73352 GB avail
  9. 91/4348880 objects degraded (0.002%)
  10. 1004 active+clean
  11. 13 active+recovery_wait
  12. 1 active+clean+scrubbing
  13. 6 active+recovering
  14. client io 254 MB/s wr, 63 op/s
  15.  
  16.  
  17. 2014-05-02 17:15:28.314480 osd.1 [WRN] slow request 31.403914 seconds old, received at 2014-05-02 17:14:56.910503: osd_op(client.4402.1:130283 1000000080c.000002d0 [write 0~4194304 [1@-1],startsync 0~0] 0.2a87d2eb RETRY=1 snapc 1=[] e235) v4 currently waiting for subops from [6]
  18. 2014-05-02 17:15:28.861571 mon.0 [INF] pgmap v71867: 1024 pgs: 1004 active+clean, 13 active+recovery_wait, 1 active+clean+scrubbing, 6 active+recovering; 8493 GB data, 17009 GB used, 56342 GB / 73352 GB avail; 254 MB/s wr, 63 op/s; 91/4348880 objects degraded (0.002%)
  19. 2014-05-02 17:15:20.384650 osd.5 [WRN] 23 slow requests, 6 included below; oldest blocked for > 37.974678 secs
  20. 2014-05-02 17:15:20.384659 osd.5 [WRN] slow request 32.000768 seconds old, received at 2014-05-02 17:14:48.383773: osd_sub_op(client.4402.1:130150 0.192 4c046b92/1000000080c.0000024b/head//0 [] v 241'4221 snapset=0=[]:[] snapc=0=[]) v8 currently commit sent
  21. 2014-05-02 17:15:20.384666 osd.5 [WRN] slow request 32.624814 seconds old, received at 2014-05-02 17:14:47.759727: osd_sub_op(client.4399.1:131574 0.1ee bfb4b1ee/10000000c04.000004fb/head//0 [] v 240'4206 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  22. 2014-05-02 17:15:20.384672 osd.5 [WRN] slow request 31.937984 seconds old, received at 2014-05-02 17:14:48.446557: osd_sub_op(client.4405.1:128201 0.118 2c89e318/10000000fde.00000238/head//0 [] v 242'4210 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  23. 2014-05-02 17:15:20.384677 osd.5 [WRN] slow request 31.745872 seconds old, received at 2014-05-02 17:14:48.638669: osd_sub_op(client.4402.1:130476 0.1d4 d9c833d4/1000000080c.00000391/head//0 [] v 240'4205 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  24. 2014-05-02 17:15:20.384683 osd.5 [WRN] slow request 31.635853 seconds old, received at 2014-05-02 17:14:48.748688: osd_op(client.4405.1:128719 10000000fde.0000043e [write 0~4194304 [1@-1],startsync 0~0] 0.6eb573c3 snapc 1=[] e234) v4 currently waiting for subops from [0]
  25. 2014-05-02 17:15:20.632218 osd.0 [WRN] 5 slow requests, 1 included below; oldest blocked for > 70.234363 secs
  26. 2014-05-02 17:15:20.632234 osd.0 [WRN] slow request 30.538392 seconds old, received at 2014-05-02 17:14:50.093702: osd_op(client.4350.1:875612 1000000010a.000013b1 [write 0~4194304 [1@-1],startsync 0~0] 0.f06caedd snapc 1=[] e234) v4 currently waiting for subops from [2]
  27. 2014-05-02 17:15:21.384951 osd.5 [WRN] 26 slow requests, 6 included below; oldest blocked for > 37.920015 secs
  28. 2014-05-02 17:15:21.384961 osd.5 [WRN] slow request 32.616188 seconds old, received at 2014-05-02 17:14:48.768659: osd_op(client.4402.1:130505 1000000080c.000003ae [write 0~4194304 [1@-1],startsync 0~0] 0.50293b0e snapc 1=[] e234) v4 currently waiting for subops from [0]
  29. 2014-05-02 17:15:21.384967 osd.5 [WRN] slow request 32.482331 seconds old, received at 2014-05-02 17:14:48.902516: osd_op(client.4402.1:130506 1000000080c.000003af [write 0~4194304 [1@-1],startsync 0~0] 0.38efa1c3 snapc 1=[] e234) v4 currently waiting for subops from [0]
  30. 2014-05-02 17:15:21.384973 osd.5 [WRN] slow request 32.568627 seconds old, received at 2014-05-02 17:14:48.816220: osd_sub_op(client.4402.1:130499 0.153 33dd1353/1000000080c.000003a8/head//0 [] v 240'4366 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  31. 2014-05-02 17:15:21.384978 osd.5 [WRN] slow request 32.883653 seconds old, received at 2014-05-02 17:14:48.501194: osd_sub_op(client.4405.1:128217 0.192 76ac4f92/10000000fde.00000248/head//0 [] v 241'4222 snapset=0=[]:[] snapc=0=[]) v8 currently commit sent
  32. 2014-05-02 17:15:21.384983 osd.5 [WRN] slow request 32.590651 seconds old, received at 2014-05-02 17:14:48.794196: osd_sub_op(client.4350.1:875604 0.8f 24f14a8f/1000000010a.000013a9/head//0 [] v 242'4281 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  33. 2014-05-02 17:15:21.632437 osd.0 [WRN] 7 slow requests, 2 included below; oldest blocked for > 71.234643 secs
  34. 2014-05-02 17:15:21.632450 osd.0 [WRN] slow request 30.680076 seconds old, received at 2014-05-02 17:14:50.952298: osd_op(client.4350.1:875616 1000000010a.000013b5 [write 0~4194304 [1@-1],startsync 0~0] 0.aecc9e43 snapc 1=[] e234) v4 currently waiting for subops from [5]
  35. 2014-05-02 17:15:21.632458 osd.0 [WRN] slow request 30.170438 seconds old, received at 2014-05-02 17:14:51.461936: osd_op(client.4350.1:875625 1000000010a.000013be [write 0~4194304 [1@-1],startsync 0~0] 0.ee1f3dad snapc 1=[] e234) v4 currently waiting for subops from [2]
  36. 2014-05-02 17:15:22.385282 osd.5 [WRN] 29 slow requests, 6 included below; oldest blocked for > 38.846325 secs
  37. 2014-05-02 17:15:22.385289 osd.5 [WRN] slow request 32.736709 seconds old, received at 2014-05-02 17:14:49.648487: osd_sub_op(client.4405.1:128221 0.105 e3ee4705/10000000fde.0000024c/head//0 [] v 242'4253 snapset=0=[]:[] snapc=0=[]) v8 currently commit sent
  38. 2014-05-02 17:15:22.385294 osd.5 [WRN] slow request 32.244398 seconds old, received at 2014-05-02 17:14:50.140798: osd_sub_op(client.4399.1:130920 0.1da 73fd9da/10000000c04.0000026d/head//0 [] v 242'4246 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  39. 2014-05-02 17:15:22.385299 osd.5 [WRN] slow request 32.186786 seconds old, received at 2014-05-02 17:14:50.198410: osd_sub_op(client.4402.1:130198 0.10f 44c710f/1000000080c.0000027b/head//0 [] v 242'4584 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  40. 2014-05-02 17:15:22.385304 osd.5 [WRN] slow request 31.955453 seconds old, received at 2014-05-02 17:14:50.429743: osd_sub_op(client.4405.1:128253 0.105 d1e46b05/10000000fde.0000026c/head//0 [] v 242'4254 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  41. 2014-05-02 17:15:22.385312 osd.5 [WRN] slow request 32.649695 seconds old, received at 2014-05-02 17:14:49.735501: osd_sub_op(client.4350.1:875608 0.d9 f5cc36d9/1000000010a.000013ad/head//0 [] v 242'4197 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  42. 2014-05-02 17:15:23.385495 osd.5 [WRN] 34 slow requests, 6 included below; oldest blocked for > 39.846547 secs
  43. 2014-05-02 17:15:23.385503 osd.5 [WRN] slow request 32.763590 seconds old, received at 2014-05-02 17:14:50.621828: osd_sub_op(client.4399.1:130928 0.10f bd23850f/10000000c04.00000275/head//0 [] v 242'4585 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  44. 2014-05-02 17:15:23.385508 osd.5 [WRN] slow request 32.550725 seconds old, received at 2014-05-02 17:14:50.834693: osd_sub_op(client.4350.1:875380 0.c5 128476c5/1000000010a.000012c9/head//0 [] v 242'4225 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  45. 2014-05-02 17:15:23.385514 osd.5 [WRN] slow request 32.653439 seconds old, received at 2014-05-02 17:14:50.731979: osd_sub_op(client.4353.1:920308 0.118 29fadb18/100000004c9.0000020a/head//0 [] v 242'4211 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  46. 2014-05-02 17:15:23.385519 osd.5 [WRN] slow request 32.415753 seconds old, received at 2014-05-02 17:14:50.969665: osd_sub_op(client.4353.1:920345 0.10f b0ca130f/100000004c9.0000022f/head//0 [] v 242'4586 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  47. 2014-05-02 17:15:23.385524 osd.5 [WRN] slow request 33.777522 seconds old, received at 2014-05-02 17:14:49.607896: osd_sub_op(client.4402.1:130528 0.a5 db3edaa5/1000000080c.000003c5/head//0 [] v 241'4235 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  48. 2014-05-02 17:15:24.385776 osd.5 [WRN] 32 slow requests, 6 included below; oldest blocked for > 40.778987 secs
  49. 2014-05-02 17:15:24.385784 osd.5 [WRN] slow request 32.540464 seconds old, received at 2014-05-02 17:14:51.845237: osd_sub_op(client.4402.1:130561 0.52 5e95d052/1000000080c.000003e6/head//0 [] v 242'4264 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  50. 2014-05-02 17:15:24.385807 osd.5 [WRN] slow request 32.437801 seconds old, received at 2014-05-02 17:14:51.947900: osd_sub_op(client.4350.1:875453 0.179 4e373379/1000000010a.00001312/head//0 [] v 242'4303 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  51. 2014-05-02 17:15:24.385813 osd.5 [WRN] slow request 32.325985 seconds old, received at 2014-05-02 17:14:52.059716: osd_sub_op(client.4402.1:130590 0.179 43468379/1000000080c.00000403/head//0 [] v 242'4304 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  52. 2014-05-02 17:15:24.385818 osd.5 [WRN] slow request 32.200077 seconds old, received at 2014-05-02 17:14:52.185624: osd_sub_op(client.4405.1:128496 0.52 7a4bb652/10000000fde.0000035f/head//0 [] v 242'4265 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  53. 2014-05-02 17:15:24.385823 osd.5 [WRN] slow request 32.154071 seconds old, received at 2014-05-02 17:14:52.231630: osd_sub_op(client.4399.1:131688 0.1df ef0ccbdf/10000000c04.0000056d/head//0 [] v 242'4236 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  54. 2014-05-02 17:15:25.386060 osd.5 [WRN] 32 slow requests, 6 included below; oldest blocked for > 39.907371 secs
  55. 2014-05-02 17:15:25.386103 osd.5 [WRN] slow request 33.107160 seconds old, received at 2014-05-02 17:14:52.278812: osd_sub_op(client.4350.1:875478 0.1d4 ecd1c3d4/1000000010a.0000132b/head//0 [] v 242'4206 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  56. 2014-05-02 17:15:25.386108 osd.5 [WRN] slow request 33.050261 seconds old, received at 2014-05-02 17:14:52.335711: osd_sub_op(client.4350.1:875534 0.a5 d20a54a5/1000000010a.00001363/head//0 [] v 242'4236 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  57. 2014-05-02 17:15:25.386114 osd.5 [WRN] slow request 32.963092 seconds old, received at 2014-05-02 17:14:52.422880: osd_sub_op(client.4350.1:875538 0.1df 3cc09ddf/1000000010a.00001367/head//0 [] v 242'4237 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  58. 2014-05-02 17:15:25.386118 osd.5 [WRN] slow request 33.934059 seconds old, received at 2014-05-02 17:14:51.451913: osd_sub_op(client.4399.1:130942 0.b7 f4195ab7/10000000c04.00000283/head//0 [] v 242'4206 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  59. 2014-05-02 17:15:25.386124 osd.5 [WRN] slow request 32.707837 seconds old, received at 2014-05-02 17:14:52.678135: osd_sub_op(client.4353.1:920714 0.1ee d9b609ee/100000004c9.000003a0/head//0 [] v 242'4207 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  60. 2014-05-02 17:15:26.386329 osd.5 [WRN] 34 slow requests, 6 included below; oldest blocked for > 40.524191 secs
  61. 2014-05-02 17:15:26.386341 osd.5 [WRN] slow request 33.786997 seconds old, received at 2014-05-02 17:14:52.599254: osd_sub_op(client.4405.1:128282 0.1da bb4f01da/10000000fde.00000289/head//0 [] v 242'4247 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  62. 2014-05-02 17:15:26.386346 osd.5 [WRN] slow request 33.557720 seconds old, received at 2014-05-02 17:14:52.828531: osd_sub_op(client.4353.1:920376 0.1ec 469b9fec/100000004c9.0000024e/head//0 [] v 242'4238 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  63. 2014-05-02 17:15:26.386352 osd.5 [WRN] slow request 33.100300 seconds old, received at 2014-05-02 17:14:53.285951: osd_sub_op(client.4350.1:875387 0.172 9be9d172/1000000010a.000012d0/head//0 [] v 242'4335 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  64. 2014-05-02 17:15:26.386356 osd.5 [WRN] slow request 32.924011 seconds old, received at 2014-05-02 17:14:53.462240: osd_sub_op(client.4350.1:875408 0.1bc 74f897bc/1000000010a.000012e5/head//0 [] v 242'4256 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  65. 2014-05-02 17:15:26.386361 osd.5 [WRN] slow request 32.771300 seconds old, received at 2014-05-02 17:14:53.614951: osd_sub_op(client.4399.1:130999 0.1bf 1ce381bf/10000000c04.000002bc/head//0 [] v 242'4292 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  66. 2014-05-02 17:15:27.386667 osd.5 [WRN] 36 slow requests, 6 included below; oldest blocked for > 41.441978 secs
  67. 2014-05-02 17:15:27.386676 osd.5 [WRN] slow request 35.789817 seconds old, received at 2014-05-02 17:14:51.596771: osd_sub_op(client.4402.1:130243 0.c9 84d7c8c9/1000000080c.000002a8/head//0 [] v 242'4234 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  68. 2014-05-02 17:15:27.386683 osd.5 [WRN] slow request 33.642989 seconds old, received at 2014-05-02 17:14:53.743599: osd_sub_op(client.4353.1:920392 0.192 85c28792/100000004c9.0000025e/head//0 [] v 242'4223 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  69. 2014-05-02 17:15:27.386688 osd.5 [WRN] slow request 33.457625 seconds old, received at 2014-05-02 17:14:53.928963: osd_sub_op(client.4402.1:130244 0.fa 2d67e6fa/1000000080c.000002a9/head//0 [] v 242'4202 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  70. 2014-05-02 17:15:27.386694 osd.5 [WRN] slow request 33.244330 seconds old, received at 2014-05-02 17:14:54.142258: osd_sub_op(client.4402.1:130264 0.1da a721e7da/1000000080c.000002bd/head//0 [] v 242'4248 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  71. 2014-05-02 17:15:27.386700 osd.5 [WRN] slow request 33.105362 seconds old, received at 2014-05-02 17:14:54.281226: osd_sub_op(client.4405.1:128347 0.10f a4f0470f/10000000fde.000002ca/head//0 [] v 242'4587 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  72. 2014-05-02 17:15:27.633664 osd.0 [WRN] 4 slow requests, 2 included below; oldest blocked for > 77.235884 secs
  73. 2014-05-02 17:15:27.633677 osd.0 [WRN] slow request 60.313641 seconds old, received at 2014-05-02 17:14:27.319974: osd_op(mds.0.4:1686 200.00000000 [writefull 0~84] 1.844f3494 e234) v4 currently no flag points reached
  74. 2014-05-02 17:15:27.633683 osd.0 [WRN] slow request 30.400179 seconds old, received at 2014-05-02 17:14:57.233436: osd_op(client.4350.1:875672 1000000010a.000013ed [write 0~4194304 [1@-1],startsync 0~0] 0.6cb07815 snapc 1=[] e234) v4 currently waiting for subops from [2]
  75. 2014-05-02 17:15:28.386873 osd.5 [WRN] 38 slow requests, 6 included below; oldest blocked for > 41.530070 secs
  76. 2014-05-02 17:15:28.386883 osd.5 [WRN] slow request 33.742715 seconds old, received at 2014-05-02 17:14:54.644081: osd_sub_op(client.4402.1:130268 0.10f a3db050f/1000000080c.000002c1/head//0 [] v 242'4588 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  77. 2014-05-02 17:15:28.386889 osd.5 [WRN] slow request 35.419887 seconds old, received at 2014-05-02 17:14:52.966909: osd_sub_op(client.4399.1:131821 0.1ee 62beefee/10000000c04.000005f2/head//0 [] v 242'4208 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  78. 2014-05-02 17:15:28.386895 osd.5 [WRN] slow request 33.151435 seconds old, received at 2014-05-02 17:14:55.235361: osd_sub_op(client.4350.1:875558 0.1ee bd2981ee/1000000010a.0000137b/head//0 [] v 242'4209 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  79. 2014-05-02 17:15:28.386899 osd.5 [WRN] slow request 33.466893 seconds old, received at 2014-05-02 17:14:54.919903: osd_sub_op(client.4399.1:131076 0.116 58538116/10000000c04.00000309/head//0 [] v 242'4330 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  80. 2014-05-02 17:15:28.386904 osd.5 [WRN] slow request 33.047457 seconds old, received at 2014-05-02 17:14:55.339339: osd_sub_op(client.4402.1:130661 0.1d4 8eb0cfd4/1000000080c.0000044a/head//0 [] v 242'4207 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  81. 2014-05-02 17:15:29.387126 osd.5 [WRN] 42 slow requests, 6 included below; oldest blocked for > 42.530298 secs
  82. 2014-05-02 17:15:29.387137 osd.5 [WRN] slow request 34.040070 seconds old, received at 2014-05-02 17:14:55.346954: osd_sub_op(client.4402.1:130270 0.192 b3796592/1000000080c.000002c3/head//0 [] v 242'4224 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  83. 2014-05-02 17:15:29.387142 osd.5 [WRN] slow request 33.975591 seconds old, received at 2014-05-02 17:14:55.411433: osd_sub_op(client.4399.1:131847 0.179 2de90379/10000000c04.0000060c/head//0 [] v 242'4305 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  84. 2014-05-02 17:15:29.387147 osd.5 [WRN] slow request 33.896490 seconds old, received at 2014-05-02 17:14:55.490534: osd_sub_op(client.4405.1:128605 0.1df 331747df/10000000fde.000003cc/head//0 [] v 242'4238 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  85. 2014-05-02 17:15:29.387152 osd.5 [WRN] slow request 33.813033 seconds old, received at 2014-05-02 17:14:55.573991: osd_sub_op(client.4399.1:131862 0.1df 835397df/10000000c04.0000061b/head//0 [] v 242'4239 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  86. 2014-05-02 17:15:29.387157 osd.5 [WRN] slow request 35.239203 seconds old, received at 2014-05-02 17:14:54.147821: osd_sub_op(client.4350.1:875644 0.88 685df488/1000000010a.000013d1/head//0 [] v 242'4230 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  87. 2014-05-02 17:15:21.303900 osd.7 [WRN] 12 slow requests, 2 included below; oldest blocked for > 38.233638 secs
  88. 2014-05-02 17:15:21.303909 osd.7 [WRN] slow request 30.159404 seconds old, received at 2014-05-02 17:14:51.144444: osd_op(client.4402.1:130244 1000000080c.000002a9 [write 0~4194304 [1@-1],startsync 0~0] 0.2d67e6fa snapc 1=[] e234) v4 currently waiting for subops from [5]
  89. 2014-05-02 17:15:21.303934 osd.7 [WRN] slow request 30.158367 seconds old, received at 2014-05-02 17:14:51.145481: osd_op(client.4350.1:875644 1000000010a.000013d1 [write 0~4194304 [1@-1],startsync 0~0] 0.685df488 snapc 1=[] e234) v4 currently waiting for subops from [5]
  90. 2014-05-02 17:15:21.492684 osd.6 [WRN] 12 slow requests, 4 included below; oldest blocked for > 84.604359 secs
  91. 2014-05-02 17:15:21.492727 osd.6 [WRN] slow request 30.501945 seconds old, received at 2014-05-02 17:14:50.990658: osd_sub_op(client.4399.1:130953 0.1f4 fb86cbf4/10000000c04.0000028e/head//0 [] v 238'4257 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  92. 2014-05-02 17:15:21.492732 osd.6 [WRN] slow request 30.474612 seconds old, received at 2014-05-02 17:14:51.017991: osd_op(client.4353.1:920589 100000004c9.00000323 [write 0~4194304 [1@-1],startsync 0~0] 0.28f64aee snapc 1=[] e234) v4 currently waiting for subops from [0]
  93. 2014-05-02 17:15:21.492737 osd.6 [WRN] slow request 30.320889 seconds old, received at 2014-05-02 17:14:51.171714: osd_op(client.4350.1:875515 1000000010a.00001350 [write 0~4194304 [1@-1],startsync 0~0] 0.b09c4558 snapc 1=[] e234) v4 currently waiting for subops from [5]
  94. 2014-05-02 17:15:21.492741 osd.6 [WRN] slow request 30.144138 seconds old, received at 2014-05-02 17:14:51.348465: osd_sub_op(client.4399.1:130962 0.59 c9977459/10000000c04.00000297/head//0 [] v 238'4294 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  95. 2014-05-02 17:15:22.304086 osd.7 [WRN] 12 slow requests, 1 included below; oldest blocked for > 39.360866 secs
  96. 2014-05-02 17:15:22.304094 osd.7 [WRN] slow request 30.750786 seconds old, received at 2014-05-02 17:14:51.553257: osd_op(client.4353.1:920377 100000004c9.0000024f [write 0~4194304 [1@-1],startsync 0~0] 0.f2f390c9 RETRY=1 snapc 1=[] e235) v4 currently waiting for subops from [5]
  97. 2014-05-02 17:15:22.493041 osd.6 [WRN] 14 slow requests, 5 included below; oldest blocked for > 85.604691 secs
  98. 2014-05-02 17:15:22.493049 osd.6 [WRN] slow request 30.324889 seconds old, received at 2014-05-02 17:14:52.168046: osd_sub_op(client.4405.1:128261 0.40 8ce56a40/10000000fde.00000274/head//0 [] v 240'4486 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  99. 2014-05-02 17:15:22.493061 osd.6 [WRN] slow request 30.195082 seconds old, received at 2014-05-02 17:14:52.297853: osd_sub_op(client.4399.1:130989 0.f2 8892b4f2/10000000c04.000002b2/head//0 [] v 242'4257 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  100. 2014-05-02 17:15:22.493068 osd.6 [WRN] slow request 30.428246 seconds old, received at 2014-05-02 17:14:52.064689: osd_op(client.4350.1:875521 1000000010a.00001356 [write 0~4194304 [1@-1],startsync 0~0] 0.a9ab0c39 snapc 1=[] e234) v4 currently waiting for subops from [5]
  101. 2014-05-02 17:15:22.493074 osd.6 [WRN] slow request 30.502390 seconds old, received at 2014-05-02 17:14:51.990545: osd_op(client.4353.1:920692 100000004c9.0000038a [write 0~4194304 [1@-1],startsync 0~0] 0.e8b5ae50 snapc 1=[] e234) v4 currently waiting for subops from [0]
  102. 2014-05-02 17:15:22.493079 osd.6 [WRN] slow request 30.095874 seconds old, received at 2014-05-02 17:14:52.397061: osd_sub_op(client.4350.1:875356 0.97 d1ee5a97/1000000010a.000012b1/head//0 [] v 242'4228 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  103. 2014-05-02 17:15:23.493420 osd.6 [WRN] 13 slow requests, 1 included below; oldest blocked for > 86.605129 secs
  104. 2014-05-02 17:15:23.493428 osd.6 [WRN] slow request 30.321141 seconds old, received at 2014-05-02 17:14:53.172232: osd_op(client.4350.1:875532 1000000010a.00001361 [write 0~4194304 [1@-1],startsync 0~0] 0.2bec1e48 snapc 1=[] e234) v4 currently commit sent
  105. 2014-05-02 17:15:24.304484 osd.7 [WRN] 13 slow requests, 2 included below; oldest blocked for > 40.935474 secs
  106. 2014-05-02 17:15:24.304495 osd.7 [WRN] slow request 30.929588 seconds old, received at 2014-05-02 17:14:53.374828: osd_op(client.4399.1:130987 10000000c04.000002b0 [write 0~4194304 [1@-1],startsync 0~0] 0.b317424b snapc 1=[] e234) v4 currently waiting for subops from [5]
  107. 2014-05-02 17:15:24.304501 osd.7 [WRN] slow request 30.391062 seconds old, received at 2014-05-02 17:14:53.913354: osd_op(client.4402.1:130251 1000000080c.000002b0 [write 0~4194304 [1@-1],startsync 0~0] 0.8b260118 snapc 1=[] e234) v4 currently waiting for subops from [5]
  108. 2014-05-02 17:15:24.493634 osd.6 [WRN] 16 slow requests, 3 included below; oldest blocked for > 87.605319 secs
  109. 2014-05-02 17:15:24.493641 osd.6 [WRN] slow request 30.935656 seconds old, received at 2014-05-02 17:14:53.557907: osd_sub_op(client.4402.1:130071 0.f2 b701f8f2/1000000080c.000001fc/head//0 [] v 242'4258 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  110. 2014-05-02 17:15:24.493648 osd.6 [WRN] slow request 31.667962 seconds old, received at 2014-05-02 17:14:52.825601: osd_op(client.4353.1:920705 100000004c9.00000397 [write 0~4194304 [1@-1],startsync 0~0] 0.eb1d0831 snapc 1=[] e234) v4 currently waiting for subops from [1]
  111. 2014-05-02 17:15:24.493653 osd.6 [WRN] slow request 30.719100 seconds old, received at 2014-05-02 17:14:53.774463: osd_op(client.4350.1:875547 1000000010a.00001370 [write 0~4194304 [1@-1],startsync 0~0] 0.a7efed7a snapc 1=[] e234) v4 currently waiting for subops from [4]
  112. 2014-05-02 17:15:25.304796 osd.7 [WRN] 12 slow requests, 1 included below; oldest blocked for > 41.264652 secs
  113. 2014-05-02 17:15:25.304831 osd.7 [WRN] slow request 30.252775 seconds old, received at 2014-05-02 17:14:55.051945: osd_op(client.4399.1:130991 10000000c04.000002b4 [write 0~4194304 [1@-1],startsync 0~0] 0.d5a96a44 snapc 1=[] e234) v4 currently waiting for subops from [5]
  114. 2014-05-02 17:15:25.494015 osd.6 [WRN] 12 slow requests, 1 included below; oldest blocked for > 88.605638 secs
  115. 2014-05-02 17:15:25.494029 osd.6 [WRN] slow request 30.961065 seconds old, received at 2014-05-02 17:14:54.532817: osd_op(client.4350.1:875553 1000000010a.00001376 [write 0~4194304 [1@-1],startsync 0~0] 0.2957de5c snapc 1=[] e234) v4 currently commit sent
  116. 2014-05-02 17:15:26.494323 osd.6 [WRN] 15 slow requests, 6 included below; oldest blocked for > 89.605994 secs
  117. 2014-05-02 17:15:26.494331 osd.6 [WRN] slow request 30.580157 seconds old, received at 2014-05-02 17:14:55.914081: osd_sub_op(client.4399.1:131022 0.97 bc139697/10000000c04.000002d3/head//0 [] v 242'4229 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  118. 2014-05-02 17:15:26.494336 osd.6 [WRN] slow request 30.477141 seconds old, received at 2014-05-02 17:14:56.017097: osd_sub_op(client.4402.1:130111 0.f2 756b6af2/1000000080c.00000224/head//0 [] v 242'4259 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  119. 2014-05-02 17:15:26.494341 osd.6 [WRN] slow request 30.403897 seconds old, received at 2014-05-02 17:14:56.090341: osd_sub_op(client.4350.1:875407 0.97 21303297/1000000010a.000012e4/head//0 [] v 242'4230 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  120. 2014-05-02 17:15:26.494347 osd.6 [WRN] slow request 31.519940 seconds old, received at 2014-05-02 17:14:54.974298: osd_op(client.4350.1:875562 1000000010a.0000137f [write 0~4194304 [1@-1],startsync 0~0] 0.cfe14c78 snapc 1=[] e234) v4 currently waiting for subops from [9]
  121. 2014-05-02 17:15:26.494362 osd.6 [WRN] slow request 30.286748 seconds old, received at 2014-05-02 17:14:56.207490: osd_op(client.4350.1:875567 1000000010a.00001384 [write 0~4194304 [1@-1],startsync 0~0] 0.c8533e50 snapc 1=[] e234) v4 currently waiting for subops from [0]
  122. 2014-05-02 17:15:27.305222 osd.7 [WRN] 12 slow requests, 2 included below; oldest blocked for > 41.474246 secs
  123. 2014-05-02 17:15:27.305230 osd.7 [WRN] slow request 30.138231 seconds old, received at 2014-05-02 17:14:57.166919: osd_op(client.4399.1:130994 10000000c04.000002b7 [write 0~4194304 [1@-1],startsync 0~0] 0.5caea2fa snapc 1=[] e234) v4 currently waiting for subops from [5]
  124. 2014-05-02 17:15:27.305237 osd.7 [WRN] slow request 31.364099 seconds old, received at 2014-05-02 17:14:55.941051: osd_op(client.4402.1:130257 1000000080c.000002b6 [write 0~4194304 [1@-1],startsync 0~0] 0.37214df9 snapc 1=[] e234) v4 currently waiting for subops from [5]
  125. 2014-05-02 17:15:27.494655 osd.6 [WRN] 16 slow requests, 4 included below; oldest blocked for > 90.606303 secs
  126. 2014-05-02 17:15:27.494662 osd.6 [WRN] slow request 31.301899 seconds old, received at 2014-05-02 17:14:56.192648: osd_sub_op(client.4402.1:130141 0.82 21d43e82/1000000080c.00000242/head//0 [] v 242'4259 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  127. 2014-05-02 17:15:27.494669 osd.6 [WRN] slow request 30.652091 seconds old, received at 2014-05-02 17:14:56.842456: osd_sub_op(client.4350.1:875410 0.1f4 e17e2bf4/1000000010a.000012e7/head//0 [] v 242'4258 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  128. 2014-05-02 17:15:27.494674 osd.6 [WRN] slow request 30.503374 seconds old, received at 2014-05-02 17:14:56.991173: osd_sub_op(client.4353.1:920680 0.97 f566da97/100000004c9.0000037e/head//0 [] v 242'4231 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  129. 2014-05-02 17:15:27.494679 osd.6 [WRN] slow request 30.080643 seconds old, received at 2014-05-02 17:14:57.413904: osd_sub_op(client.4399.1:131096 0.40 caf40a40/10000000c04.0000031d/head//0 [] v 242'4487 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  130. 2014-05-02 17:15:28.495058 osd.6 [WRN] 14 slow requests, 1 included below; oldest blocked for > 91.606740 secs
  131. 2014-05-02 17:15:28.495065 osd.6 [WRN] slow request 32.204947 seconds old, received at 2014-05-02 17:14:56.290037: osd_op(client.4350.1:875570 1000000010a.00001387 [write 0~4194304 [1@-1],startsync 0~0] 0.dff6f58e snapc 1=[] e234) v4 currently commit sent
  132. 2014-05-02 17:15:29.778765 mon.0 [INF] pgmap v71868: 1024 pgs: 1004 active+clean, 13 active+recovery_wait, 1 active+clean+scrubbing, 6 active+recovering; 8493 GB data, 17009 GB used, 56342 GB / 73352 GB avail; 171 MB/s wr, 42 op/s; 91/4348914 objects degraded (0.002%)
  133. 2014-05-02 17:15:30.305698 osd.7 [WRN] 12 slow requests, 2 included below; oldest blocked for > 43.190854 secs
  134. 2014-05-02 17:15:30.305708 osd.7 [WRN] slow request 30.790983 seconds old, received at 2014-05-02 17:14:59.514652: osd_op(client.4405.1:128273 10000000fde.00000280 [write 0~4194304 [1@-1],startsync 0~0] 0.b1437ed9 snapc 1=[] e234) v4 currently waiting for subops from [5]
  135. 2014-05-02 17:15:30.305713 osd.7 [WRN] slow request 30.434707 seconds old, received at 2014-05-02 17:14:59.870928: osd_op(client.4399.1:131015 10000000c04.000002cc [write 0~4194304 [1@-1],startsync 0~0] 0.32870fb5 snapc 1=[] e234) v4 currently waiting for subops from [5]
  136. 2014-05-02 17:15:30.495465 osd.6 [WRN] 14 slow requests, 2 included below; oldest blocked for > 93.607150 secs
  137. 2014-05-02 17:15:30.495473 osd.6 [WRN] slow request 30.481036 seconds old, received at 2014-05-02 17:15:00.014358: osd_op(client.4350.1:875575 1000000010a.0000138c [write 0~4194304 [1@-1],startsync 0~0] 0.50eeab56 snapc 1=[] e234) v4 currently waiting for subops from [4]
  138. 2014-05-02 17:15:30.495478 osd.6 [WRN] slow request 31.427114 seconds old, received at 2014-05-02 17:14:59.068280: osd_sub_op(client.4402.1:130189 0.f2 de55caf2/1000000080c.00000272/head//0 [] v 242'4260 snapset=0=[]:[] snapc=0=[]) v8 currently no flag points reached
  139. 2014-05-02 17:15:31.647631 mon.0 [INF] pgmap v71869: 1024 pgs: 1004 active+clean, 13 active+recovery_wait, 1 active+clean+scrubbing, 6 active+recovering; 8493 GB data, 17009 GB used, 56342 GB / 73352 GB avail; 124 MB/s wr, 31 op/s; 97/4349020 objects degraded (0.002%)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!