API tools faq
paste
Advertisement
Guest User

log

a guest
Apr 22nd, 2013
45
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 64.11 KB | None | 0 0
raw download clone embed print report
  1. 2013-04-22 07:56:32.795550 7fd848e38700 0 log [INF] : 1.106 deep-scrub ok
  2. 2013-04-22 07:57:10.276846 7fd848e38700 0 log [INF] : 2.4f deep-scrub ok
  3. 2013-04-22 08:25:59.515253 7fd848e38700 0 log [INF] : 1.11b deep-scrub ok
  4. 2013-04-22 08:26:57.047844 7fd848e38700 0 log [INF] : 2.50 deep-scrub ok
  5. 2013-04-22 09:12:23.266907 7fd848e38700 0 log [INF] : 2.55 deep-scrub ok
  6. 2013-04-22 09:24:17.171021 7fd84f645700 -1 FileStore: sync_entry timed out after 600 seconds.
  7. ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca)
  8. 1: (SafeTimer::timer_thread()+0x425) [0x826925]
  9. 2: (SafeTimerThread::entry()+0xd) [0x82756d]
  10. 3: (()+0x7e9a) [0x7fd85b48fe9a]
  11. 4: (clone()+0x6d) [0x7fd859efbcbd]
  12.  
  13. 2013-04-22 09:24:17.197331 7fd84f645700 -1 os/FileStore.cc: In function 'virtual void SyncEntryTimeout::finish(int)' thread 7fd84f645700 time 2013-04-22 09:24:17.195485
  14. os/FileStore.cc: 3298: FAILED assert(0)
  15.  
  16. ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca)
  17. 1: (SyncEntryTimeout::finish(int)+0x9e) [0x720b7e]
  18. 2: (SafeTimer::timer_thread()+0x425) [0x826925]
  19. 3: (SafeTimerThread::entry()+0xd) [0x82756d]
  20. 4: (()+0x7e9a) [0x7fd85b48fe9a]
  21. 5: (clone()+0x6d) [0x7fd859efbcbd]
  22. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  23.  
  24. --- begin dump of recent events ---
  25. -10000> 2013-04-22 09:22:59.344981 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.344818, event: all_read, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  26. -9999> 2013-04-22 09:22:59.345005 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.344912, event: dispatched, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  27. -9998> 2013-04-22 09:22:59.345031 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.345030, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  28. -9997> 2013-04-22 09:22:59.345106 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.345105, event: reached_pg, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  29. -9996> 2013-04-22 09:22:59.361709 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524258 ==== osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4 ==== 170+0+0 (3424628950 0 0) 0x6b15d80 con 0x645d000
  30. -9995> 2013-04-22 09:22:59.361783 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361531, event: header_read, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  31. -9994> 2013-04-22 09:22:59.361820 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361538, event: throttled, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  32. -9993> 2013-04-22 09:22:59.361845 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361670, event: all_read, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  33. -9992> 2013-04-22 09:22:59.361868 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361773, event: dispatched, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  34. -9991> 2013-04-22 09:22:59.361893 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361893, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  35. -9990> 2013-04-22 09:22:59.361966 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.361965, event: reached_pg, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  36. -9989> 2013-04-22 09:22:59.362146 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.362146, event: started, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  37. -9988> 2013-04-22 09:22:59.362458 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] = 0) v4 -- ?+0 0x40dc400 con 0x645d000
  38. -9987> 2013-04-22 09:22:59.362504 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876194, seq: 27619457, time: 2013-04-22 09:22:59.362504, event: done, request: osd_op(client.650315.0:27876194 rb.0.15ad.238e1f29.000000000679 [sparse-read 3682304~131072] 2.27091637) v4
  39. -9986> 2013-04-22 09:22:59.364257 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.364257, event: started, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  40. -9985> 2013-04-22 09:22:59.399417 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] = 0) v4 -- ?+0 0x5a3c200 con 0x645d000
  41. -9984> 2013-04-22 09:22:59.399496 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876193, seq: 27619456, time: 2013-04-22 09:22:59.399495, event: done, request: osd_op(client.650315.0:27876193 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 0~61440] 2.5135426e) v4
  42. -9983> 2013-04-22 09:22:59.434649 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524259 ==== osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4 ==== 170+0+0 (3653557636 0 0) 0x390c900 con 0x645d000
  43. -9982> 2013-04-22 09:22:59.434714 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434452, event: header_read, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  44. -9981> 2013-04-22 09:22:59.434749 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434458, event: throttled, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  45. -9980> 2013-04-22 09:22:59.434773 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434613, event: all_read, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  46. -9979> 2013-04-22 09:22:59.434797 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434705, event: dispatched, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  47. -9978> 2013-04-22 09:22:59.434822 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434822, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  48. -9977> 2013-04-22 09:22:59.434900 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.434899, event: reached_pg, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  49. -9976> 2013-04-22 09:22:59.435082 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.435082, event: started, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  50. -9975> 2013-04-22 09:22:59.435399 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] = 0) v4 -- ?+0 0x602e400 con 0x645d000
  51. -9974> 2013-04-22 09:22:59.435445 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876195, seq: 27619458, time: 2013-04-22 09:22:59.435444, event: done, request: osd_op(client.650315.0:27876195 rb.0.15ad.238e1f29.000000000679 [sparse-read 3813376~131072] 2.27091637) v4
  52. -9973> 2013-04-22 09:22:59.475114 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524260 ==== osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4 ==== 170+0+0 (3886639730 0 0) 0x390c480 con 0x645d000
  53. -9972> 2013-04-22 09:22:59.475177 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.474953, event: header_read, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  54. -9971> 2013-04-22 09:22:59.475212 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.474959, event: throttled, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  55. -9970> 2013-04-22 09:22:59.475237 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475076, event: all_read, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  56. -9969> 2013-04-22 09:22:59.475261 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475167, event: dispatched, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  57. -9968> 2013-04-22 09:22:59.475287 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475287, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  58. -9967> 2013-04-22 09:22:59.475364 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475363, event: reached_pg, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  59. -9966> 2013-04-22 09:22:59.475546 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475545, event: started, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  60. -9965> 2013-04-22 09:22:59.475839 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] = 0) v4 -- ?+0 0x69c3800 con 0x645d000
  61. -9964> 2013-04-22 09:22:59.475886 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876196, seq: 27619459, time: 2013-04-22 09:22:59.475886, event: done, request: osd_op(client.650315.0:27876196 rb.0.15ad.238e1f29.000000000679 [sparse-read 3944448~131072] 2.27091637) v4
  62. -9963> 2013-04-22 09:22:59.544778 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524261 ==== osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4 ==== 170+0+0 (2734133521 0 0) 0x5f5dd80 con 0x645d000
  63. -9962> 2013-04-22 09:22:59.544842 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.544615, event: header_read, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  64. -9961> 2013-04-22 09:22:59.544878 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.544620, event: throttled, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  65. -9960> 2013-04-22 09:22:59.544902 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.544738, event: all_read, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  66. -9959> 2013-04-22 09:22:59.544926 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.544833, event: dispatched, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  67. -9958> 2013-04-22 09:22:59.544951 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.544951, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  68. -9957> 2013-04-22 09:22:59.545025 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.545025, event: reached_pg, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  69. -9956> 2013-04-22 09:22:59.545204 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.545204, event: started, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  70. -9955> 2013-04-22 09:22:59.545811 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] = 0) v4 -- ?+0 0x60b4200 con 0x645d000
  71. -9954> 2013-04-22 09:22:59.545863 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876200, seq: 27619460, time: 2013-04-22 09:22:59.545863, event: done, request: osd_op(client.650315.0:27876200 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 61440~131072] 2.5135426e) v4
  72. -9953> 2013-04-22 09:22:59.546944 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524262 ==== osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4 ==== 170+0+0 (1737038050 0 0) 0x5f85000 con 0x645d000
  73. -9952> 2013-04-22 09:22:59.546993 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.546632, event: header_read, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  74. -9951> 2013-04-22 09:22:59.547036 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.546637, event: throttled, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  75. -9950> 2013-04-22 09:22:59.547083 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.546791, event: all_read, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  76. -9949> 2013-04-22 09:22:59.547106 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.546984, event: dispatched, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  77. -9948> 2013-04-22 09:22:59.547132 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.547131, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  78. -9947> 2013-04-22 09:22:59.547195 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524263 ==== osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4 ==== 170+0+0 (2740728918 0 0) 0x20c9900 con 0x645d000
  79. -9946> 2013-04-22 09:22:59.547216 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.547215, event: reached_pg, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  80. -9945> 2013-04-22 09:22:59.547282 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.546823, event: header_read, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  81. -9944> 2013-04-22 09:22:59.547304 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.546825, event: throttled, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  82. -9943> 2013-04-22 09:22:59.547319 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.546890, event: all_read, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  83. -9942> 2013-04-22 09:22:59.547351 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.547219, event: dispatched, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  84. -9941> 2013-04-22 09:22:59.547367 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.547367, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  85. -9940> 2013-04-22 09:22:59.547430 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.547429, event: reached_pg, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  86. -9939> 2013-04-22 09:22:59.547469 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.547435, event: started, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  87. -9938> 2013-04-22 09:22:59.547694 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] = 0) v4 -- ?+0 0x5cc4a00 con 0x645d000
  88. -9937> 2013-04-22 09:22:59.547732 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.547732, event: started, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  89. -9936> 2013-04-22 09:22:59.547775 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876201, seq: 27619461, time: 2013-04-22 09:22:59.547775, event: done, request: osd_op(client.650315.0:27876201 rb.0.15ad.238e1f29.000000000679 [sparse-read 4075520~118784] 2.27091637) v4
  90. -9935> 2013-04-22 09:22:59.571336 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524264 ==== osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4 ==== 170+0+0 (3827328081 0 0) 0x63b2480 con 0x645d000
  91. -9934> 2013-04-22 09:22:59.571410 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571160, event: header_read, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  92. -9933> 2013-04-22 09:22:59.571446 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571166, event: throttled, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  93. -9932> 2013-04-22 09:22:59.571470 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571297, event: all_read, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  94. -9931> 2013-04-22 09:22:59.571493 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571399, event: dispatched, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  95. -9930> 2013-04-22 09:22:59.571512 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571512, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  96. -9929> 2013-04-22 09:22:59.571589 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571588, event: reached_pg, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  97. -9928> 2013-04-22 09:22:59.571768 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.571768, event: started, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  98. -9927> 2013-04-22 09:22:59.572080 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] = 0) v4 -- ?+0 0x6aa8600 con 0x645d000
  99. -9926> 2013-04-22 09:22:59.572126 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876203, seq: 27619463, time: 2013-04-22 09:22:59.572126, event: done, request: osd_op(client.650315.0:27876203 rb.0.15ad.238e1f29.0000000016df [sparse-read 1699840~131072] 2.7ad16475) v4
  100. -9925> 2013-04-22 09:22:59.580510 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] = 0) v4 -- ?+0 0x602e200 con 0x645d000
  101. -9924> 2013-04-22 09:22:59.580590 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876202, seq: 27619462, time: 2013-04-22 09:22:59.580589, event: done, request: osd_op(client.650315.0:27876202 rb.0.15ad.238e1f29.00000000067a [sparse-read 0~12288] 2.9263bbd7) v4
  102. -9923> 2013-04-22 09:22:59.585090 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524265 ==== osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4 ==== 170+0+0 (3156503553 0 0) 0x5df8240 con 0x645d000
  103. -9922> 2013-04-22 09:22:59.585154 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.584929, event: header_read, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  104. -9921> 2013-04-22 09:22:59.585188 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.584935, event: throttled, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  105. -9920> 2013-04-22 09:22:59.585212 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.585051, event: all_read, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  106. -9919> 2013-04-22 09:22:59.585235 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.585144, event: dispatched, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  107. -9918> 2013-04-22 09:22:59.585260 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.585260, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  108. -9917> 2013-04-22 09:22:59.585334 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.585334, event: reached_pg, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  109. -9916> 2013-04-22 09:22:59.585540 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.585539, event: started, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  110. -9915> 2013-04-22 09:22:59.598233 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] = 0) v4 -- ?+0 0x6a22200 con 0x645d000
  111. -9914> 2013-04-22 09:22:59.598314 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876204, seq: 27619464, time: 2013-04-22 09:22:59.598313, event: done, request: osd_op(client.650315.0:27876204 rb.0.15ad.238e1f29.00000000067a [sparse-read 12288~131072] 2.9263bbd7) v4
  112. -9913> 2013-04-22 09:22:59.656198 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524266 ==== osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4 ==== 170+0+0 (2852061523 0 0) 0x63c2b40 con 0x645d000
  113. -9912> 2013-04-22 09:22:59.656262 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656037, event: header_read, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  114. -9911> 2013-04-22 09:22:59.656298 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656042, event: throttled, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  115. -9910> 2013-04-22 09:22:59.656323 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656159, event: all_read, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  116. -9909> 2013-04-22 09:22:59.656346 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656253, event: dispatched, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  117. -9908> 2013-04-22 09:22:59.656372 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656372, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  118. -9907> 2013-04-22 09:22:59.656448 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656448, event: reached_pg, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  119. -9906> 2013-04-22 09:22:59.656654 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.656654, event: started, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  120. -9905> 2013-04-22 09:22:59.656987 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] = 0) v4 -- ?+0 0x6232800 con 0x645d000
  121. -9904> 2013-04-22 09:22:59.657034 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876207, seq: 27619465, time: 2013-04-22 09:22:59.657033, event: done, request: osd_op(client.650315.0:27876207 rb.0.15ad.238e1f29.00000000067a [sparse-read 143360~131072] 2.9263bbd7) v4
  122. -9903> 2013-04-22 09:22:59.666024 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524267 ==== osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4 ==== 170+0+0 (373215531 0 0) 0xd0036c0 con 0x645d000
  123. -9902> 2013-04-22 09:22:59.666088 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.665861, event: header_read, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  124. -9901> 2013-04-22 09:22:59.666124 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.665866, event: throttled, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  125. -9900> 2013-04-22 09:22:59.666148 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.665984, event: all_read, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  126. -9899> 2013-04-22 09:22:59.666173 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.666079, event: dispatched, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  127. -9898> 2013-04-22 09:22:59.666197 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.666197, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  128. -9897> 2013-04-22 09:22:59.666274 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.666274, event: reached_pg, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  129. -9896> 2013-04-22 09:22:59.666454 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.666454, event: started, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  130. -9895> 2013-04-22 09:22:59.666811 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] = 0) v4 -- ?+0 0x595fa00 con 0x645d000
  131. -9894> 2013-04-22 09:22:59.666857 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876208, seq: 27619466, time: 2013-04-22 09:22:59.666856, event: done, request: osd_op(client.650315.0:27876208 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1847296~131072] 2.fca37a6c) v4
  132. -9893> 2013-04-22 09:22:59.719581 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524268 ==== osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4 ==== 170+0+0 (3073178691 0 0) 0xc8a6240 con 0x645d000
  133. -9892> 2013-04-22 09:22:59.719645 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719412, event: header_read, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  134. -9891> 2013-04-22 09:22:59.719681 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719417, event: throttled, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  135. -9890> 2013-04-22 09:22:59.719705 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719541, event: all_read, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  136. -9889> 2013-04-22 09:22:59.719729 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719636, event: dispatched, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  137. -9888> 2013-04-22 09:22:59.719754 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719754, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  138. -9887> 2013-04-22 09:22:59.719831 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.719831, event: reached_pg, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  139. -9886> 2013-04-22 09:22:59.720011 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.720011, event: started, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  140. -9885> 2013-04-22 09:22:59.720324 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] = 0) v4 -- ?+0 0x5a3c800 con 0x645d000
  141. -9884> 2013-04-22 09:22:59.720372 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876210, seq: 27619467, time: 2013-04-22 09:22:59.720371, event: done, request: osd_op(client.650315.0:27876210 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 192512~131072] 2.5135426e) v4
  142. -9883> 2013-04-22 09:22:59.733462 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524269 ==== osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4 ==== 170+0+0 (2542974629 0 0) 0x5c95d80 con 0x645d000
  143. -9882> 2013-04-22 09:22:59.733529 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733300, event: header_read, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  144. -9881> 2013-04-22 09:22:59.733564 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733305, event: throttled, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  145. -9880> 2013-04-22 09:22:59.733588 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733422, event: all_read, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  146. -9879> 2013-04-22 09:22:59.733611 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733519, event: dispatched, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  147. -9878> 2013-04-22 09:22:59.733636 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733635, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  148. -9877> 2013-04-22 09:22:59.733711 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733711, event: reached_pg, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  149. -9876> 2013-04-22 09:22:59.733924 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.733924, event: started, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  150. -9875> 2013-04-22 09:22:59.734259 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] = 0) v4 -- ?+0 0x60ed200 con 0x645d000
  151. -9874> 2013-04-22 09:22:59.734306 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876211, seq: 27619468, time: 2013-04-22 09:22:59.734306, event: done, request: osd_op(client.650315.0:27876211 rb.0.15ad.238e1f29.00000000067a [sparse-read 274432~131072] 2.9263bbd7) v4
  152. -9873> 2013-04-22 09:22:59.766954 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524270 ==== osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4 ==== 170+0+0 (2185878519 0 0) 0x59e96c0 con 0x645d000
  153. -9872> 2013-04-22 09:22:59.767020 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.766761, event: header_read, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  154. -9871> 2013-04-22 09:22:59.767056 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.766766, event: throttled, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  155. -9870> 2013-04-22 09:22:59.767079 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.766883, event: all_read, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  156. -9869> 2013-04-22 09:22:59.767101 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.767011, event: dispatched, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  157. -9868> 2013-04-22 09:22:59.767127 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.767127, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  158. -9867> 2013-04-22 09:22:59.767216 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.767215, event: reached_pg, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  159. -9866> 2013-04-22 09:22:59.767423 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.767423, event: started, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  160. -9865> 2013-04-22 09:22:59.767766 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] = 0) v4 -- ?+0 0x60ede00 con 0x645d000
  161. -9864> 2013-04-22 09:22:59.767811 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876212, seq: 27619469, time: 2013-04-22 09:22:59.767810, event: done, request: osd_op(client.650315.0:27876212 rb.0.15ad.238e1f29.00000000067a [sparse-read 405504~131072] 2.9263bbd7) v4
  162. -9863> 2013-04-22 09:22:59.805287 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524271 ==== osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4 ==== 170+0+0 (3947485513 0 0) 0x59e9480 con 0x645d000
  163. -9862> 2013-04-22 09:22:59.805358 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805125, event: header_read, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  164. -9861> 2013-04-22 09:22:59.805393 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805131, event: throttled, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  165. -9860> 2013-04-22 09:22:59.805417 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805249, event: all_read, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  166. -9859> 2013-04-22 09:22:59.805439 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805348, event: dispatched, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  167. -9858> 2013-04-22 09:22:59.805464 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805464, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  168. -9857> 2013-04-22 09:22:59.805540 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805539, event: reached_pg, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  169. -9856> 2013-04-22 09:22:59.805745 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.805744, event: started, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  170. -9855> 2013-04-22 09:22:59.806079 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] = 0) v4 -- ?+0 0x57a7c00 con 0x645d000
  171. -9854> 2013-04-22 09:22:59.806126 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876214, seq: 27619470, time: 2013-04-22 09:22:59.806126, event: done, request: osd_op(client.650315.0:27876214 rb.0.15ad.238e1f29.00000000067a [sparse-read 536576~131072] 2.9263bbd7) v4
  172. -9853> 2013-04-22 09:22:59.863099 7fd856e54700 5 osd.5 1381 tick
  173. -9852> 2013-04-22 09:22:59.879057 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524272 ==== osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4 ==== 170+0+0 (4271027227 0 0) 0x2fb0000 con 0x645d000
  174. -9851> 2013-04-22 09:22:59.879127 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.878897, event: header_read, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  175. -9850> 2013-04-22 09:22:59.879162 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.878903, event: throttled, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  176. -9849> 2013-04-22 09:22:59.879186 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879019, event: all_read, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  177. -9848> 2013-04-22 09:22:59.879211 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879117, event: dispatched, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  178. -9847> 2013-04-22 09:22:59.879236 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879236, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  179. -9846> 2013-04-22 09:22:59.879321 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879321, event: reached_pg, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  180. -9845> 2013-04-22 09:22:59.879527 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879527, event: started, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  181. -9844> 2013-04-22 09:22:59.879866 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] = 0) v4 -- ?+0 0x6004400 con 0x645d000
  182. -9843> 2013-04-22 09:22:59.879913 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876216, seq: 27619471, time: 2013-04-22 09:22:59.879912, event: done, request: osd_op(client.650315.0:27876216 rb.0.15ad.238e1f29.00000000067a [sparse-read 667648~131072] 2.9263bbd7) v4
  183. -9842> 2013-04-22 09:22:59.919841 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524273 ==== osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4 ==== 170+0+0 (2302757813 0 0) 0x5a0cd80 con 0x645d000
  184. -9841> 2013-04-22 09:22:59.919961 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.919690, event: header_read, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  185. -9840> 2013-04-22 09:22:59.919990 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.919696, event: throttled, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  186. -9839> 2013-04-22 09:22:59.920007 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.919808, event: all_read, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  187. -9838> 2013-04-22 09:22:59.920023 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.919884, event: dispatched, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  188. -9837> 2013-04-22 09:22:59.920039 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.920039, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  189. -9836> 2013-04-22 09:22:59.920104 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.920103, event: reached_pg, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  190. -9835> 2013-04-22 09:22:59.920278 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.920278, event: started, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  191. -9834> 2013-04-22 09:22:59.920584 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] = 0) v4 -- ?+0 0x32c9a00 con 0x645d000
  192. -9833> 2013-04-22 09:22:59.920627 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876219, seq: 27619472, time: 2013-04-22 09:22:59.920626, event: done, request: osd_op(client.650315.0:27876219 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 323584~131072] 2.5135426e) v4
  193. -9832> 2013-04-22 09:22:59.951908 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524274 ==== osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4 ==== 170+0+0 (3237938157 0 0) 0x5a0c240 con 0x645d000
  194. -9831> 2013-04-22 09:22:59.951978 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.951739, event: header_read, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  195. -9830> 2013-04-22 09:22:59.952013 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.951744, event: throttled, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  196. -9829> 2013-04-22 09:22:59.952037 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.951870, event: all_read, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  197. -9828> 2013-04-22 09:22:59.952061 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.951968, event: dispatched, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  198. -9827> 2013-04-22 09:22:59.952086 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.952085, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  199. -9826> 2013-04-22 09:22:59.952163 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.952163, event: reached_pg, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  200. -9825> 2013-04-22 09:22:59.952368 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.952368, event: started, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  201. -9824> 2013-04-22 09:22:59.952705 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] = 0) v4 -- ?+0 0x5e8fe00 con 0x645d000
  202. -9823> 2013-04-22 09:22:59.952753 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876220, seq: 27619473, time: 2013-04-22 09:22:59.952752, event: done, request: osd_op(client.650315.0:27876220 rb.0.15ad.238e1f29.00000000067a [sparse-read 798720~131072] 2.9263bbd7) v4
  203. -9822> 2013-04-22 09:22:59.991139 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524275 ==== osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4 ==== 170+0+0 (3575934655 0 0) 0x6042900 con 0x645d000
  204. -9821> 2013-04-22 09:22:59.991208 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.990974, event: header_read, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  205. -9820> 2013-04-22 09:22:59.991243 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.990979, event: throttled, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  206. -9819> 2013-04-22 09:22:59.991268 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991097, event: all_read, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  207. -9818> 2013-04-22 09:22:59.991291 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991198, event: dispatched, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  208. -9817> 2013-04-22 09:22:59.991315 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991315, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  209. -9816> 2013-04-22 09:22:59.991391 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991391, event: reached_pg, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  210. -9815> 2013-04-22 09:22:59.991596 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991596, event: started, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  211. -9814> 2013-04-22 09:22:59.991927 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] = 0) v4 -- ?+0 0x427bc00 con 0x645d000
  212. -9813> 2013-04-22 09:22:59.991973 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876222, seq: 27619474, time: 2013-04-22 09:22:59.991972, event: done, request: osd_op(client.650315.0:27876222 rb.0.15ad.238e1f29.00000000067a [sparse-read 929792~131072] 2.9263bbd7) v4
  213. -9812> 2013-04-22 09:23:00.043813 7fd847e36700 5 osd.5 1381 heartbeat: osd_stat(152 GB used, 1709 GB avail, 1862 GB total, peers [0,1,2,3]/[])
  214. -9811> 2013-04-22 09:23:00.043860 7fd847e36700 1 -- 10.90.90.9:0/1130 --> 10.90.90.7:6802/980 -- osd_ping(ping e1381 stamp 2013-04-22 09:23:00.043855) v2 -- ?+0 0x5f25700 con 0x1d0b420
  215. -9810> 2013-04-22 09:23:00.043919 7fd847e36700 1 -- 10.90.90.9:0/1130 --> 10.90.90.7:6805/1072 -- osd_ping(ping e1381 stamp 2013-04-22 09:23:00.043855) v2 -- ?+0 0x5b62380 con 0x1d0b2c0
  216. -9809> 2013-04-22 09:23:00.043949 7fd847e36700 1 -- 10.90.90.9:0/1130 --> 10.90.90.8:6802/1040 -- osd_ping(ping e1381 stamp 2013-04-22 09:23:00.043855) v2 -- ?+0 0x5a9d540 con 0x589ec60
  217. -9808> 2013-04-22 09:23:00.043975 7fd847e36700 1 -- 10.90.90.9:0/1130 --> 10.90.90.8:6805/1129 -- osd_ping(ping e1381 stamp 2013-04-22 09:23:00.043855) v2 -- ?+0 0x6a821c0 con 0x589eb00
  218. -9807> 2013-04-22 09:23:00.045011 7fd84ce40700 1 -- 10.90.90.9:0/1130 <== osd.1 10.90.90.7:6805/1072 498202 ==== osd_ping(ping_reply e1381 stamp 2013-04-22 09:23:00.043855) v2 ==== 47+0+0 (2831228620 0 0) 0x2b4ce00 con 0x1d0b2c0
  219. -9806> 2013-04-22 09:23:00.045125 7fd84ce40700 1 -- 10.90.90.9:0/1130 <== osd.3 10.90.90.8:6805/1129 498204 ==== osd_ping(ping_reply e1381 stamp 2013-04-22 09:23:00.043855) v2 ==== 47+0+0 (2831228620 0 0) 0x674a380 con 0x589eb00
  220. -9805> 2013-04-22 09:23:00.045157 7fd84ce40700 1 -- 10.90.90.9:0/1130 <== osd.0 10.90.90.7:6802/980 498204 ==== osd_ping(ping_reply e1381 stamp 2013-04-22 09:23:00.043855) v2 ==== 47+0+0 (2831228620 0 0) 0x5ac88c0 con 0x1d0b420
  221. -9804> 2013-04-22 09:23:00.045187 7fd84ce40700 1 -- 10.90.90.9:0/1130 <== osd.2 10.90.90.8:6802/1040 498204 ==== osd_ping(ping_reply e1381 stamp 2013-04-22 09:23:00.043855) v2 ==== 47+0+0 (2831228620 0 0) 0x5fa7540 con 0x589ec60
  222. -9803> 2013-04-22 09:23:00.045249 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524276 ==== osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4 ==== 170+0+0 (2627348199 0 0) 0x6042b40 con 0x645d000
  223. -9802> 2013-04-22 09:23:00.045289 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045105, event: header_read, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  224. -9801> 2013-04-22 09:23:00.045313 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045108, event: throttled, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  225. -9800> 2013-04-22 09:23:00.045335 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045224, event: all_read, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  226. -9799> 2013-04-22 09:23:00.045357 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045280, event: dispatched, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  227. -9798> 2013-04-22 09:23:00.045380 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045380, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  228. -9797> 2013-04-22 09:23:00.045453 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045453, event: reached_pg, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  229. -9796> 2013-04-22 09:23:00.045635 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045634, event: started, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  230. -9795> 2013-04-22 09:23:00.045949 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] = 0) v4 -- ?+0 0x6004c00 con 0x645d000
  231. -9794> 2013-04-22 09:23:00.045995 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876224, seq: 27619475, time: 2013-04-22 09:23:00.045994, event: done, request: osd_op(client.650315.0:27876224 rb.0.15ad.238e1f29.000000000cb2 [sparse-read 454656~131072] 2.5135426e) v4
  232. -9793> 2013-04-22 09:23:00.063213 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524277 ==== osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4 ==== 170+0+0 (318703249 0 0) 0x6042000 con 0x645d000
  233. -9792> 2013-04-22 09:23:00.063283 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063054, event: header_read, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  234. -9791> 2013-04-22 09:23:00.063319 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063059, event: throttled, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  235. -9790> 2013-04-22 09:23:00.063343 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063176, event: all_read, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  236. -9789> 2013-04-22 09:23:00.063367 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063273, event: dispatched, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  237. -9788> 2013-04-22 09:23:00.063392 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063391, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  238. -9787> 2013-04-22 09:23:00.063466 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063466, event: reached_pg, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  239. -9786> 2013-04-22 09:23:00.063670 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.063669, event: started, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  240. -9785> 2013-04-22 09:23:00.064003 7fd84a63b700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] = 0) v4 -- ?+0 0x6172a00 con 0x645d000
  241. -9784> 2013-04-22 09:23:00.064052 7fd84a63b700 5 --OSD::tracker-- reqid: client.650315.0:27876225, seq: 27619476, time: 2013-04-22 09:23:00.064051, event: done, request: osd_op(client.650315.0:27876225 rb.0.15ad.238e1f29.00000000067a [sparse-read 1060864~131072] 2.9263bbd7) v4
  242. -9783> 2013-04-22 09:23:00.065647 7fd84ee44700 1 -- 10.90.90.9:6803/1130 <== client.650315 10.90.90.5:0/1896885 524278 ==== osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4 ==== 170+0+0 (65387641 0 0) 0x6042240 con 0x645d000
  243. -9782> 2013-04-22 09:23:00.065707 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065531, event: header_read, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  244. -9781> 2013-04-22 09:23:00.065738 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065541, event: throttled, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  245. -9780> 2013-04-22 09:23:00.065762 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065614, event: all_read, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  246. -9779> 2013-04-22 09:23:00.065785 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065698, event: dispatched, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  247. -9778> 2013-04-22 09:23:00.065810 7fd84ee44700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065810, event: waiting_for_osdmap, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  248. -9777> 2013-04-22 09:23:00.065886 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.065886, event: reached_pg, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  249. -9776> 2013-04-22 09:23:00.066056 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.066055, event: started, request: osd_op(client.650315.0:27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] 2.fca37a6c) v4
  250. -9775> 2013-04-22 09:23:00.066365 7fd849e3a700 1 -- 10.90.90.9:6803/1130 --> 10.90.90.5:0/1896885 -- osd_op_reply(27876226 rb.0.15ad.238e1f29.0000000013b7 [sparse-read 1978368~131072] = 0) v4 -- ?+0 0x32c9200 con 0x645d000
  251. -9774> 2013-04-22 09:23:00.066407 7fd849e3a700 5 --OSD::tracker-- reqid: client.650315.0:27876226, seq: 27619477, time: 2013-04-22 09:23:00.066407, event: done, request: osd_op(client.650315.0:27876226
  252.  
  253.  
  254.  
  255. =============================
  256.  
  257.  
  258. cat /var/log/ceph/ceph.log | grep osd.5
  259. 2013-04-22 07:56:32.795554 osd.5 10.90.90.9:6803/1130 1689 : [INF] 1.106 deep-scrub ok
  260. 2013-04-22 07:57:10.276849 osd.5 10.90.90.9:6803/1130 1690 : [INF] 2.4f deep-scrub ok
  261. 2013-04-22 08:25:59.515256 osd.5 10.90.90.9:6803/1130 1691 : [INF] 1.11b deep-scrub ok
  262. 2013-04-22 08:26:57.047848 osd.5 10.90.90.9:6803/1130 1692 : [INF] 2.50 deep-scrub ok
  263. 2013-04-22 09:12:23.266912 osd.5 10.90.90.9:6803/1130 1693 : [INF] 2.55 deep-scrub ok
  264. 2013-04-22 09:24:46.844317 mon.0 10.90.90.7:6789/0 1351646 : [DBG] osd.5 10.90.90.9:6803/1130 reported failed by osd.3 10.90.90.8:6803/1129
  265. 2013-04-22 09:24:51.730302 mon.0 10.90.90.7:6789/0 1351651 : [DBG] osd.5 10.90.90.9:6803/1130 reported failed by osd.2 10.90.90.8:6800/1040
  266. 2013-04-22 09:24:51.779931 mon.0 10.90.90.7:6789/0 1351652 : [DBG] osd.5 10.90.90.9:6803/1130 reported failed by osd.3 10.90.90.8:6803/1129
  267. 2013-04-22 09:24:51.780155 mon.0 10.90.90.7:6789/0 1351653 : [INF] osd.5 10.90.90.9:6803/1130 failed (3 reports from 2 peers after 2013-04-22 09:25:11.779907 >= grace 20.000000)
  268. 2013-04-22 09:29:55.962627 mon.0 10.90.90.7:6789/0 1351907 : [INF] osd.5 out (down for 303.687060)
  269. 2013-04-22 11:01:54.736777 mon.0 10.90.90.7:6789/0 1356291 : [INF] osd.5 10.90.90.9:6803/1102 boot
  270. 2013-04-22 11:01:59.809347 osd.5 10.90.90.9:6803/1102 1 : [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.353984 secs
  271. 2013-04-22 11:01:59.809362 osd.5 10.90.90.9:6803/1102 2 : [WRN] slow request 30.353984 seconds old, received at 2013-04-22 11:01:29.455256: osd_op(client.701989.0:8158742 rbd_header.ab6ec2ae8944a [watch 1~0] 2.a590337) v4 currently reached pg
  272. 2013-04-22 11:02:08.690193 osd.5 10.90.90.9:6803/1102 3 : [WRN] 2 slow requests, 1 included below; oldest blocked for > 39.234900 secs
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!