API tools faq
paste
Advertisement
alohamora

osd.118

alohamora
Jun 2nd, 2014
62
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 59.34 KB | None | 0 0
raw download clone embed print report
  1. -369> 2014-06-02 11:09:00.678422 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.558258, event: header_read, request: pg_info(1 pgs e10842:16.d79) v4
  2. -368> 2014-06-02 11:09:00.678426 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.558259, event: throttled, request: pg_info(1 pgs e10842:16.d79) v4
  3. -367> 2014-06-02 11:09:00.678429 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.558316, event: all_read, request: pg_info(1 pgs e10842:16.d79) v4
  4. -366> 2014-06-02 11:09:00.678432 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.678417, event: dispatched, request: pg_info(1 pgs e10842:16.d79) v4
  5. -365> 2014-06-02 11:09:00.678436 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.678436, event: waiting_for_osdmap, request: pg_info(1 pgs e10842:16.d79) v4
  6. -364> 2014-06-02 11:09:00.678439 7f98392d2700 7 osd.118 10842 handle_pg_info pg_info(1 pgs e10842:16.d79) v4 from osd.46
  7. -363> 2014-06-02 11:09:00.678441 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0xd692f40
  8. -362> 2014-06-02 11:09:00.678444 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.678444, event: started, request: pg_info(1 pgs e10842:16.d79) v4
  9. -361> 2014-06-02 11:09:00.678457 7f98392d2700 5 -- op tracker -- , seq: 804709, time: 2014-06-02 11:09:00.678456, event: done, request: pg_info(1 pgs e10842:16.d79) v4
  10. -360> 2014-06-02 11:09:00.678465 7f98392d2700 10 osd.118 10842 do_waiters -- start
  11. -359> 2014-06-02 11:09:00.678469 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  12. -358> 2014-06-02 11:09:00.678474 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.158 192.168.1.113:6809/84049479 16 ==== pg_notify(16.121(1) epoch 10842) v5 ==== 769+0+0 (1834866590 0 0) 0x9045780 con 0x11908c60
  13. -357> 2014-06-02 11:09:00.678482 7f98392d2700 10 osd.118 10842 do_waiters -- start
  14. -356> 2014-06-02 11:09:00.678483 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  15. -355> 2014-06-02 11:09:00.678484 7f98392d2700 20 osd.118 10842 _dispatch 0x9045780 pg_notify(16.121(1) epoch 10842) v5
  16. -354> 2014-06-02 11:09:00.678487 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.664098, event: header_read, request: pg_notify(16.121(1) epoch 10842) v5
  17. -353> 2014-06-02 11:09:00.678491 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.664100, event: throttled, request: pg_notify(16.121(1) epoch 10842) v5
  18. -352> 2014-06-02 11:09:00.678494 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.664181, event: all_read, request: pg_notify(16.121(1) epoch 10842) v5
  19. -351> 2014-06-02 11:09:00.678497 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.678482, event: dispatched, request: pg_notify(16.121(1) epoch 10842) v5
  20. -350> 2014-06-02 11:09:00.678500 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.678500, event: waiting_for_osdmap, request: pg_notify(16.121(1) epoch 10842) v5
  21. -349> 2014-06-02 11:09:00.678506 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.158
  22. -348> 2014-06-02 11:09:00.678508 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x9045780
  23. -347> 2014-06-02 11:09:00.678510 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.678510, event: started, request: pg_notify(16.121(1) epoch 10842) v5
  24. -346> 2014-06-02 11:09:00.678521 7f98392d2700 5 -- op tracker -- , seq: 804710, time: 2014-06-02 11:09:00.678521, event: done, request: pg_notify(16.121(1) epoch 10842) v5
  25. -345> 2014-06-02 11:09:00.678529 7f98392d2700 10 osd.118 10842 do_waiters -- start
  26. -344> 2014-06-02 11:09:00.678533 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  27. -343> 2014-06-02 11:09:00.678539 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.113 192.168.1.113:6800/15058041 16 ==== pg_notify(10.19(7) epoch 10842) v5 ==== 1083+0+0 (132556027 0 0) 0xd691a40 con 0x119091e0
  28. -342> 2014-06-02 11:09:00.678547 7f98392d2700 10 osd.118 10842 do_waiters -- start
  29. -341> 2014-06-02 11:09:00.678549 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  30. -340> 2014-06-02 11:09:00.678550 7f98392d2700 20 osd.118 10842 _dispatch 0xd691a40 pg_notify(10.19(7) epoch 10842) v5
  31. -339> 2014-06-02 11:09:00.678553 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.629463, event: header_read, request: pg_notify(10.19(7) epoch 10842) v5
  32. -338> 2014-06-02 11:09:00.678557 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.629465, event: throttled, request: pg_notify(10.19(7) epoch 10842) v5
  33. -337> 2014-06-02 11:09:00.678560 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.629533, event: all_read, request: pg_notify(10.19(7) epoch 10842) v5
  34. -336> 2014-06-02 11:09:00.678563 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.678547, event: dispatched, request: pg_notify(10.19(7) epoch 10842) v5
  35. -335> 2014-06-02 11:09:00.678567 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.678567, event: waiting_for_osdmap, request: pg_notify(10.19(7) epoch 10842) v5
  36. -334> 2014-06-02 11:09:00.678572 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.113
  37. -333> 2014-06-02 11:09:00.678574 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0xd691a40
  38. -332> 2014-06-02 11:09:00.678577 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.678576, event: started, request: pg_notify(10.19(7) epoch 10842) v5
  39. -331> 2014-06-02 11:09:00.678588 7f98392d2700 5 -- op tracker -- , seq: 804711, time: 2014-06-02 11:09:00.678588, event: done, request: pg_notify(10.19(7) epoch 10842) v5
  40. -330> 2014-06-02 11:09:00.678596 7f98392d2700 10 osd.118 10842 do_waiters -- start
  41. -329> 2014-06-02 11:09:00.678600 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  42. -328> 2014-06-02 11:09:00.678607 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.68 192.168.1.108:6824/97011928 21 ==== pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (1908503546 0 0) 0x85ca400 con 0x198aa100
  43. -327> 2014-06-02 11:09:00.678616 7f98392d2700 10 osd.118 10842 do_waiters -- start
  44. -326> 2014-06-02 11:09:00.678619 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  45. -325> 2014-06-02 11:09:00.678622 7f98392d2700 20 osd.118 10842 _dispatch 0x85ca400 pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  46. -324> 2014-06-02 11:09:00.678626 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.399055, event: header_read, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  47. -323> 2014-06-02 11:09:00.678630 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.399056, event: throttled, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  48. -322> 2014-06-02 11:09:00.678635 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.399231, event: all_read, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  49. -321> 2014-06-02 11:09:00.678639 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.678616, event: dispatched, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  50. -320> 2014-06-02 11:09:00.678646 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.678646, event: waiting_for_osdmap, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  51. -319> 2014-06-02 11:09:00.678652 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.68
  52. -318> 2014-06-02 11:09:00.678658 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x85ca400
  53. -317> 2014-06-02 11:09:00.678660 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.678660, event: started, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  54. -316> 2014-06-02 11:09:00.678669 7f98392d2700 5 -- op tracker -- , seq: 804712, time: 2014-06-02 11:09:00.678669, event: done, request: pg_log(12.326 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  55. -315> 2014-06-02 11:09:00.678679 7f98392d2700 10 osd.118 10842 do_waiters -- start
  56. -314> 2014-06-02 11:09:00.678682 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  57. -313> 2014-06-02 11:09:00.678687 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.104 192.168.1.109:6808/10032986 16 ==== pg_info(1 pgs e10842:0.15a(1)) v4 ==== 765+0+0 (2198934457 0 0) 0xcfc3800 con 0x4f20160
  58. -312> 2014-06-02 11:09:00.678695 7f98392d2700 10 osd.118 10842 do_waiters -- start
  59. -311> 2014-06-02 11:09:00.678696 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  60. -310> 2014-06-02 11:09:00.678697 7f98392d2700 20 osd.118 10842 _dispatch 0xcfc3800 pg_info(1 pgs e10842:0.15a(1)) v4
  61. -309> 2014-06-02 11:09:00.678700 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.421791, event: header_read, request: pg_info(1 pgs e10842:0.15a(1)) v4
  62. -308> 2014-06-02 11:09:00.678704 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.421794, event: throttled, request: pg_info(1 pgs e10842:0.15a(1)) v4
  63. -307> 2014-06-02 11:09:00.678707 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.421875, event: all_read, request: pg_info(1 pgs e10842:0.15a(1)) v4
  64. -306> 2014-06-02 11:09:00.678711 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.678695, event: dispatched, request: pg_info(1 pgs e10842:0.15a(1)) v4
  65. -305> 2014-06-02 11:09:00.678717 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.678717, event: waiting_for_osdmap, request: pg_info(1 pgs e10842:0.15a(1)) v4
  66. -304> 2014-06-02 11:09:00.678723 7f98392d2700 7 osd.118 10842 handle_pg_info pg_info(1 pgs e10842:0.15a(1)) v4 from osd.104
  67. -303> 2014-06-02 11:09:00.678726 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0xcfc3800
  68. -302> 2014-06-02 11:09:00.678729 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.678729, event: started, request: pg_info(1 pgs e10842:0.15a(1)) v4
  69. -301> 2014-06-02 11:09:00.678741 7f98392d2700 5 -- op tracker -- , seq: 804713, time: 2014-06-02 11:09:00.678740, event: done, request: pg_info(1 pgs e10842:0.15a(1)) v4
  70. -300> 2014-06-02 11:09:00.678749 7f98392d2700 10 osd.118 10842 do_waiters -- start
  71. -299> 2014-06-02 11:09:00.678753 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  72. -298> 2014-06-02 11:09:00.678757 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.89 192.168.1.109:6827/80018984 13 ==== pg_info(1 pgs e10842:12.281) v4 ==== 714+0+0 (3450515911 0 0) 0x159a8c40 con 0x768d280
  73. -297> 2014-06-02 11:09:00.678765 7f98392d2700 10 osd.118 10842 do_waiters -- start
  74. -296> 2014-06-02 11:09:00.678766 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  75. -295> 2014-06-02 11:09:00.678767 7f98392d2700 20 osd.118 10842 _dispatch 0x159a8c40 pg_info(1 pgs e10842:12.281) v4
  76. -294> 2014-06-02 11:09:00.678770 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.520280, event: header_read, request: pg_info(1 pgs e10842:12.281) v4
  77. -293> 2014-06-02 11:09:00.678774 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.520282, event: throttled, request: pg_info(1 pgs e10842:12.281) v4
  78. -292> 2014-06-02 11:09:00.678777 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.520367, event: all_read, request: pg_info(1 pgs e10842:12.281) v4
  79. -291> 2014-06-02 11:09:00.678780 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.678764, event: dispatched, request: pg_info(1 pgs e10842:12.281) v4
  80. -290> 2014-06-02 11:09:00.678784 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.678784, event: waiting_for_osdmap, request: pg_info(1 pgs e10842:12.281) v4
  81. -289> 2014-06-02 11:09:00.678789 7f98392d2700 7 osd.118 10842 handle_pg_info pg_info(1 pgs e10842:12.281) v4 from osd.89
  82. -288> 2014-06-02 11:09:00.678792 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x159a8c40
  83. -287> 2014-06-02 11:09:00.678794 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.678794, event: started, request: pg_info(1 pgs e10842:12.281) v4
  84. -286> 2014-06-02 11:09:00.678805 7f98392d2700 5 -- op tracker -- , seq: 804714, time: 2014-06-02 11:09:00.678805, event: done, request: pg_info(1 pgs e10842:12.281) v4
  85. -285> 2014-06-02 11:09:00.678813 7f98392d2700 10 osd.118 10842 do_waiters -- start
  86. -284> 2014-06-02 11:09:00.678817 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  87. -283> 2014-06-02 11:09:00.678822 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.72 192.168.1.107:6802/10018860 20 ==== pg_info(1 pgs e10842:0.232(1)) v4 ==== 765+0+0 (277696051 0 0) 0x52be3c0 con 0x1fc70420
  88. -282> 2014-06-02 11:09:00.678830 7f98392d2700 10 osd.118 10842 do_waiters -- start
  89. -281> 2014-06-02 11:09:00.678833 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  90. -280> 2014-06-02 11:09:00.678834 7f98392d2700 20 osd.118 10842 _dispatch 0x52be3c0 pg_info(1 pgs e10842:0.232(1)) v4
  91. -279> 2014-06-02 11:09:00.678837 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.427880, event: header_read, request: pg_info(1 pgs e10842:0.232(1)) v4
  92. -278> 2014-06-02 11:09:00.678840 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.427882, event: throttled, request: pg_info(1 pgs e10842:0.232(1)) v4
  93. -277> 2014-06-02 11:09:00.678844 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.427938, event: all_read, request: pg_info(1 pgs e10842:0.232(1)) v4
  94. -276> 2014-06-02 11:09:00.678847 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.678830, event: dispatched, request: pg_info(1 pgs e10842:0.232(1)) v4
  95. -275> 2014-06-02 11:09:00.678851 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.678851, event: waiting_for_osdmap, request: pg_info(1 pgs e10842:0.232(1)) v4
  96. -274> 2014-06-02 11:09:00.678856 7f98392d2700 7 osd.118 10842 handle_pg_info pg_info(1 pgs e10842:0.232(1)) v4 from osd.72
  97. -273> 2014-06-02 11:09:00.678859 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x52be3c0
  98. -272> 2014-06-02 11:09:00.678862 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.678862, event: started, request: pg_info(1 pgs e10842:0.232(1)) v4
  99. -271> 2014-06-02 11:09:00.678874 7f98392d2700 5 -- op tracker -- , seq: 804715, time: 2014-06-02 11:09:00.678873, event: done, request: pg_info(1 pgs e10842:0.232(1)) v4
  100. -270> 2014-06-02 11:09:00.678882 7f98392d2700 10 osd.118 10842 do_waiters -- start
  101. -269> 2014-06-02 11:09:00.678886 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  102. -268> 2014-06-02 11:09:00.678897 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.65 192.168.1.109:6810/76006485 15 ==== pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (3866491544 0 0) 0x355b000 con 0x1fc761a0
  103. -267> 2014-06-02 11:09:00.678910 7f98392d2700 10 osd.118 10842 do_waiters -- start
  104. -266> 2014-06-02 11:09:00.678911 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  105. -265> 2014-06-02 11:09:00.678912 7f98392d2700 20 osd.118 10842 _dispatch 0x355b000 pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  106. -264> 2014-06-02 11:09:00.678916 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.491874, event: header_read, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  107. -263> 2014-06-02 11:09:00.678921 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.491877, event: throttled, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  108. -262> 2014-06-02 11:09:00.678925 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.491954, event: all_read, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  109. -261> 2014-06-02 11:09:00.678929 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.678909, event: dispatched, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  110. -260> 2014-06-02 11:09:00.678936 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.678936, event: waiting_for_osdmap, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  111. -259> 2014-06-02 11:09:00.678943 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.65
  112. -258> 2014-06-02 11:09:00.678949 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x355b000
  113. -257> 2014-06-02 11:09:00.678951 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.678951, event: started, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  114. -256> 2014-06-02 11:09:00.678961 7f98392d2700 5 -- op tracker -- , seq: 804716, time: 2014-06-02 11:09:00.678961, event: done, request: pg_log(11.2 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  115. -255> 2014-06-02 11:09:00.678970 7f98392d2700 10 osd.118 10842 do_waiters -- start
  116. -254> 2014-06-02 11:09:00.678974 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  117. -253> 2014-06-02 11:09:00.678978 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.5 192.168.1.101:6844/114018707 6 ==== osd_map(10842..10842 src has 3254..10842) v3 ==== 2256+0+0 (4211986158 0 0) 0x1fd2d340 con 0x7688160
  118. -252> 2014-06-02 11:09:00.678986 7f98392d2700 10 osd.118 10842 do_waiters -- start
  119. -251> 2014-06-02 11:09:00.678987 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  120. -250> 2014-06-02 11:09:00.678988 7f98392d2700 20 osd.118 10842 _dispatch 0x1fd2d340 osd_map(10842..10842 src has 3254..10842) v3
  121. -249> 2014-06-02 11:09:00.678995 7f98392d2700 3 osd.118 10842 handle_osd_map epochs [10842,10842], i have 10842, src has [3254,10842]
  122. -248> 2014-06-02 11:09:00.678998 7f98392d2700 10 osd.118 10842 no new maps here, dropping
  123. -247> 2014-06-02 11:09:00.679000 7f98392d2700 10 osd.118 10842 do_waiters -- start
  124. -246> 2014-06-02 11:09:00.679001 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  125. -245> 2014-06-02 11:09:00.679004 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.79 192.168.1.109:6820/86064224 18 ==== pg_notify(6.164(587) epoch 10842) v5 ==== 32023+0+0 (1570566865 0 0) 0x14804980 con 0x179927e0
  126. -244> 2014-06-02 11:09:00.679012 7f98392d2700 10 osd.118 10842 do_waiters -- start
  127. -243> 2014-06-02 11:09:00.679013 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  128. -242> 2014-06-02 11:09:00.679014 7f98392d2700 20 osd.118 10842 _dispatch 0x14804980 pg_notify(6.164(587) epoch 10842) v5
  129. -241> 2014-06-02 11:09:00.679017 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.522757, event: header_read, request: pg_notify(6.164(587) epoch 10842) v5
  130. -240> 2014-06-02 11:09:00.679021 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.522760, event: throttled, request: pg_notify(6.164(587) epoch 10842) v5
  131. -239> 2014-06-02 11:09:00.679024 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.523675, event: all_read, request: pg_notify(6.164(587) epoch 10842) v5
  132. -238> 2014-06-02 11:09:00.679027 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.679012, event: dispatched, request: pg_notify(6.164(587) epoch 10842) v5
  133. -237> 2014-06-02 11:09:00.679031 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.679031, event: waiting_for_osdmap, request: pg_notify(6.164(587) epoch 10842) v5
  134. -236> 2014-06-02 11:09:00.679034 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.79
  135. -235> 2014-06-02 11:09:00.679036 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x14804980
  136. -234> 2014-06-02 11:09:00.679038 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.679038, event: started, request: pg_notify(6.164(587) epoch 10842) v5
  137. -233> 2014-06-02 11:09:00.679049 7f98392d2700 5 -- op tracker -- , seq: 804717, time: 2014-06-02 11:09:00.679049, event: done, request: pg_notify(6.164(587) epoch 10842) v5
  138. -232> 2014-06-02 11:09:00.679056 7f98392d2700 10 osd.118 10842 do_waiters -- start
  139. -231> 2014-06-02 11:09:00.679060 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  140. -230> 2014-06-02 11:09:00.679159 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.122 192.168.1.111:6819/100052702 18 ==== pg_notify(7.18e(592) epoch 10842) v5 ==== 32278+0+0 (2669300493 0 0) 0x19709f80 con 0x1322eca0
  141. -229> 2014-06-02 11:09:00.679172 7f98392d2700 10 osd.118 10842 do_waiters -- start
  142. -228> 2014-06-02 11:09:00.679174 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  143. -227> 2014-06-02 11:09:00.679175 7f98392d2700 20 osd.118 10842 _dispatch 0x19709f80 pg_notify(7.18e(592) epoch 10842) v5
  144. -226> 2014-06-02 11:09:00.679178 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.455729, event: header_read, request: pg_notify(7.18e(592) epoch 10842) v5
  145. -225> 2014-06-02 11:09:00.679184 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.455731, event: throttled, request: pg_notify(7.18e(592) epoch 10842) v5
  146. -224> 2014-06-02 11:09:00.679188 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.465984, event: all_read, request: pg_notify(7.18e(592) epoch 10842) v5
  147. -223> 2014-06-02 11:09:00.679191 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.679172, event: dispatched, request: pg_notify(7.18e(592) epoch 10842) v5
  148. -222> 2014-06-02 11:09:00.679195 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.679194, event: waiting_for_osdmap, request: pg_notify(7.18e(592) epoch 10842) v5
  149. -221> 2014-06-02 11:09:00.679200 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.122
  150. -220> 2014-06-02 11:09:00.679206 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x19709f80
  151. -219> 2014-06-02 11:09:00.679209 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.679209, event: started, request: pg_notify(7.18e(592) epoch 10842) v5
  152. -218> 2014-06-02 11:09:00.679220 7f98392d2700 5 -- op tracker -- , seq: 804718, time: 2014-06-02 11:09:00.679220, event: done, request: pg_notify(7.18e(592) epoch 10842) v5
  153. -217> 2014-06-02 11:09:00.679230 7f98392d2700 10 osd.118 10842 do_waiters -- start
  154. -216> 2014-06-02 11:09:00.679233 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  155. -215> 2014-06-02 11:09:00.679359 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.96 192.168.1.107:6815/12019917 16 ==== pg_notify(0.181(1) epoch 10842) v5 ==== 769+0+0 (1461860653 0 0) 0x1d3e9180 con 0x173298c0
  156. -214> 2014-06-02 11:09:00.679372 7f98392d2700 10 osd.118 10842 do_waiters -- start
  157. -213> 2014-06-02 11:09:00.679374 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  158. -212> 2014-06-02 11:09:00.679375 7f98392d2700 20 osd.118 10842 _dispatch 0x1d3e9180 pg_notify(0.181(1) epoch 10842) v5
  159. -211> 2014-06-02 11:09:00.679378 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.508571, event: header_read, request: pg_notify(0.181(1) epoch 10842) v5
  160. -210> 2014-06-02 11:09:00.679383 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.508573, event: throttled, request: pg_notify(0.181(1) epoch 10842) v5
  161. -209> 2014-06-02 11:09:00.679386 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.508664, event: all_read, request: pg_notify(0.181(1) epoch 10842) v5
  162. -208> 2014-06-02 11:09:00.679389 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.679372, event: dispatched, request: pg_notify(0.181(1) epoch 10842) v5
  163. -207> 2014-06-02 11:09:00.679393 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.679393, event: waiting_for_osdmap, request: pg_notify(0.181(1) epoch 10842) v5
  164. -206> 2014-06-02 11:09:00.679399 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.96
  165. -205> 2014-06-02 11:09:00.679401 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x1d3e9180
  166. -204> 2014-06-02 11:09:00.679404 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.679404, event: started, request: pg_notify(0.181(1) epoch 10842) v5
  167. -203> 2014-06-02 11:09:00.679417 7f98392d2700 5 -- op tracker -- , seq: 804719, time: 2014-06-02 11:09:00.679417, event: done, request: pg_notify(0.181(1) epoch 10842) v5
  168. -202> 2014-06-02 11:09:00.679426 7f98392d2700 10 osd.118 10842 do_waiters -- start
  169. -201> 2014-06-02 11:09:00.679430 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  170. -200> 2014-06-02 11:09:00.679434 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.74 192.168.1.110:6835/16024166 16 ==== pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (3942283219 0 0) 0x131af200 con 0x167891e0
  171. -199> 2014-06-02 11:09:00.679443 7f98392d2700 10 osd.118 10842 do_waiters -- start
  172. -198> 2014-06-02 11:09:00.679444 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  173. -197> 2014-06-02 11:09:00.679446 7f98392d2700 20 osd.118 10842 _dispatch 0x131af200 pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  174. -196> 2014-06-02 11:09:00.679449 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.517165, event: header_read, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  175. -195> 2014-06-02 11:09:00.679454 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.517167, event: throttled, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  176. -194> 2014-06-02 11:09:00.679458 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.517251, event: all_read, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  177. -193> 2014-06-02 11:09:00.679462 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.679443, event: dispatched, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  178. -192> 2014-06-02 11:09:00.679470 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.679470, event: waiting_for_osdmap, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  179. -191> 2014-06-02 11:09:00.679476 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.74
  180. -190> 2014-06-02 11:09:00.679482 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x131af200
  181. -189> 2014-06-02 11:09:00.679486 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.679486, event: started, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  182. -188> 2014-06-02 11:09:00.679496 7f98392d2700 5 -- op tracker -- , seq: 804720, time: 2014-06-02 11:09:00.679496, event: done, request: pg_log(9.13a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  183. -187> 2014-06-02 11:09:00.679505 7f98392d2700 10 osd.118 10842 do_waiters -- start
  184. -186> 2014-06-02 11:09:00.679509 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  185. -185> 2014-06-02 11:09:00.679512 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.159 192.168.1.115:6815/73008493 18 ==== pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (1521415192 0 0) 0x672f200 con 0xcc8f640
  186. -184> 2014-06-02 11:09:00.679520 7f98392d2700 10 osd.118 10842 do_waiters -- start
  187. -183> 2014-06-02 11:09:00.679522 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  188. -182> 2014-06-02 11:09:00.679523 7f98392d2700 20 osd.118 10842 _dispatch 0x672f200 pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  189. -181> 2014-06-02 11:09:00.679526 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.539520, event: header_read, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  190. -180> 2014-06-02 11:09:00.679531 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.539523, event: throttled, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  191. -179> 2014-06-02 11:09:00.679536 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.539622, event: all_read, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  192. -178> 2014-06-02 11:09:00.679541 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.679520, event: dispatched, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  193. -177> 2014-06-02 11:09:00.679548 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.679548, event: waiting_for_osdmap, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  194. -176> 2014-06-02 11:09:00.679555 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.159
  195. -175> 2014-06-02 11:09:00.679560 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x672f200
  196. -174> 2014-06-02 11:09:00.679571 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.679571, event: started, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  197. -173> 2014-06-02 11:09:00.679581 7f98392d2700 5 -- op tracker -- , seq: 804721, time: 2014-06-02 11:09:00.679581, event: done, request: pg_log(16.e9a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  198. -172> 2014-06-02 11:09:00.679591 7f98392d2700 10 osd.118 10842 do_waiters -- start
  199. -171> 2014-06-02 11:09:00.679594 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  200. -170> 2014-06-02 11:09:00.679598 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.82 192.168.1.107:6814/61012714 16 ==== pg_notify(16.1e9(4) epoch 10842) v5 ==== 922+0+0 (1763256932 0 0) 0x15cf1f80 con 0xcc88c60
  201. -169> 2014-06-02 11:09:00.679606 7f98392d2700 10 osd.118 10842 do_waiters -- start
  202. -168> 2014-06-02 11:09:00.679607 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  203. -167> 2014-06-02 11:09:00.679608 7f98392d2700 20 osd.118 10842 _dispatch 0x15cf1f80 pg_notify(16.1e9(4) epoch 10842) v5
  204. -166> 2014-06-02 11:09:00.679611 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.482139, event: header_read, request: pg_notify(16.1e9(4) epoch 10842) v5
  205. -165> 2014-06-02 11:09:00.679615 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.482142, event: throttled, request: pg_notify(16.1e9(4) epoch 10842) v5
  206. -164> 2014-06-02 11:09:00.679618 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.482225, event: all_read, request: pg_notify(16.1e9(4) epoch 10842) v5
  207. -163> 2014-06-02 11:09:00.679621 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.679606, event: dispatched, request: pg_notify(16.1e9(4) epoch 10842) v5
  208. -162> 2014-06-02 11:09:00.679625 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.679625, event: waiting_for_osdmap, request: pg_notify(16.1e9(4) epoch 10842) v5
  209. -161> 2014-06-02 11:09:00.679630 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.82
  210. -160> 2014-06-02 11:09:00.679633 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x15cf1f80
  211. -159> 2014-06-02 11:09:00.679636 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.679636, event: started, request: pg_notify(16.1e9(4) epoch 10842) v5
  212. -158> 2014-06-02 11:09:00.679647 7f98392d2700 5 -- op tracker -- , seq: 804722, time: 2014-06-02 11:09:00.679647, event: done, request: pg_notify(16.1e9(4) epoch 10842) v5
  213. -157> 2014-06-02 11:09:00.679655 7f98392d2700 10 osd.118 10842 do_waiters -- start
  214. -156> 2014-06-02 11:09:00.679659 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  215. -155> 2014-06-02 11:09:00.679663 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.145 192.168.1.115:6813/74032359 18 ==== pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5 ==== 62223+0+0 (2789494302 0 0) 0x1d496900 con 0x5efd6a0
  216. -154> 2014-06-02 11:09:00.679671 7f98392d2700 10 osd.118 10842 do_waiters -- start
  217. -153> 2014-06-02 11:09:00.679673 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  218. -152> 2014-06-02 11:09:00.679674 7f98392d2700 20 osd.118 10842 _dispatch 0x1d496900 pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  219. -151> 2014-06-02 11:09:00.679677 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.572123, event: header_read, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  220. -150> 2014-06-02 11:09:00.679681 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.572124, event: throttled, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  221. -149> 2014-06-02 11:09:00.679685 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.573331, event: all_read, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  222. -148> 2014-06-02 11:09:00.679688 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.679671, event: dispatched, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  223. -147> 2014-06-02 11:09:00.679692 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.679692, event: waiting_for_osdmap, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  224. -146> 2014-06-02 11:09:00.679698 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.145
  225. -145> 2014-06-02 11:09:00.679700 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x1d496900
  226. -144> 2014-06-02 11:09:00.679703 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.679702, event: started, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  227. -143> 2014-06-02 11:09:00.679720 7f98392d2700 5 -- op tracker -- , seq: 804723, time: 2014-06-02 11:09:00.679720, event: done, request: pg_notify(1.2e7(590),0.2f1(559) epoch 10842) v5
  228. -142> 2014-06-02 11:09:00.679729 7f98392d2700 10 osd.118 10842 do_waiters -- start
  229. -141> 2014-06-02 11:09:00.679733 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  230. -140> 2014-06-02 11:09:00.679903 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.109 192.168.1.109:6825/14032740 18 ==== pg_notify(12.16c(1) epoch 10842) v5 ==== 769+0+0 (1032297416 0 0) 0x1d48fa80 con 0x1732ad60
  231. -139> 2014-06-02 11:09:00.679916 7f98392d2700 10 osd.118 10842 do_waiters -- start
  232. -138> 2014-06-02 11:09:00.679918 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  233. -137> 2014-06-02 11:09:00.679919 7f98392d2700 20 osd.118 10842 _dispatch 0x1d48fa80 pg_notify(12.16c(1) epoch 10842) v5
  234. -136> 2014-06-02 11:09:00.679922 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.554818, event: header_read, request: pg_notify(12.16c(1) epoch 10842) v5
  235. -135> 2014-06-02 11:09:00.679927 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.554820, event: throttled, request: pg_notify(12.16c(1) epoch 10842) v5
  236. -134> 2014-06-02 11:09:00.679930 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.554875, event: all_read, request: pg_notify(12.16c(1) epoch 10842) v5
  237. -133> 2014-06-02 11:09:00.679933 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.679916, event: dispatched, request: pg_notify(12.16c(1) epoch 10842) v5
  238. -132> 2014-06-02 11:09:00.679937 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.679937, event: waiting_for_osdmap, request: pg_notify(12.16c(1) epoch 10842) v5
  239. -131> 2014-06-02 11:09:00.679943 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.109
  240. -130> 2014-06-02 11:09:00.679944 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x1d48fa80
  241. -129> 2014-06-02 11:09:00.679947 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.679947, event: started, request: pg_notify(12.16c(1) epoch 10842) v5
  242. -128> 2014-06-02 11:09:00.679959 7f98392d2700 5 -- op tracker -- , seq: 804724, time: 2014-06-02 11:09:00.679959, event: done, request: pg_notify(12.16c(1) epoch 10842) v5
  243. -127> 2014-06-02 11:09:00.679968 7f98392d2700 10 osd.118 10842 do_waiters -- start
  244. -126> 2014-06-02 11:09:00.679972 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  245. -125> 2014-06-02 11:09:00.679976 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.153 192.168.1.115:6808/60063945 16 ==== pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (166384271 0 0) 0x111c7200 con 0xf801a20
  246. -124> 2014-06-02 11:09:00.679985 7f98392d2700 10 osd.118 10842 do_waiters -- start
  247. -123> 2014-06-02 11:09:00.679987 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  248. -122> 2014-06-02 11:09:00.679988 7f98392d2700 20 osd.118 10842 _dispatch 0x111c7200 pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  249. -121> 2014-06-02 11:09:00.679991 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.602057, event: header_read, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  250. -120> 2014-06-02 11:09:00.679996 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.602059, event: throttled, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  251. -119> 2014-06-02 11:09:00.680000 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.602109, event: all_read, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  252. -118> 2014-06-02 11:09:00.680004 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.679985, event: dispatched, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  253. -117> 2014-06-02 11:09:00.680009 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.680009, event: waiting_for_osdmap, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  254. -116> 2014-06-02 11:09:00.680016 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.153
  255. -115> 2014-06-02 11:09:00.680021 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x111c7200
  256. -114> 2014-06-02 11:09:00.680023 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.680023, event: started, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  257. -113> 2014-06-02 11:09:00.680032 7f98392d2700 5 -- op tracker -- , seq: 804725, time: 2014-06-02 11:09:00.680032, event: done, request: pg_log(11.149 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  258. -112> 2014-06-02 11:09:00.680042 7f98392d2700 10 osd.118 10842 do_waiters -- start
  259. -111> 2014-06-02 11:09:00.680045 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  260. -110> 2014-06-02 11:09:00.680048 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.49 192.168.1.103:6814/85036053 16 ==== pg_notify(16.feb(1) epoch 10842) v5 ==== 769+0+0 (45860697 0 0) 0xb379c00 con 0xcc8eca0
  261. -109> 2014-06-02 11:09:00.680056 7f98392d2700 10 osd.118 10842 do_waiters -- start
  262. -108> 2014-06-02 11:09:00.680057 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  263. -107> 2014-06-02 11:09:00.680058 7f98392d2700 20 osd.118 10842 _dispatch 0xb379c00 pg_notify(16.feb(1) epoch 10842) v5
  264. -106> 2014-06-02 11:09:00.680061 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.457909, event: header_read, request: pg_notify(16.feb(1) epoch 10842) v5
  265. -105> 2014-06-02 11:09:00.680065 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.457911, event: throttled, request: pg_notify(16.feb(1) epoch 10842) v5
  266. -104> 2014-06-02 11:09:00.680068 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.458001, event: all_read, request: pg_notify(16.feb(1) epoch 10842) v5
  267. -103> 2014-06-02 11:09:00.680071 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.680056, event: dispatched, request: pg_notify(16.feb(1) epoch 10842) v5
  268. -102> 2014-06-02 11:09:00.680075 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.680075, event: waiting_for_osdmap, request: pg_notify(16.feb(1) epoch 10842) v5
  269. -101> 2014-06-02 11:09:00.680081 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.49
  270. -100> 2014-06-02 11:09:00.680082 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0xb379c00
  271. -99> 2014-06-02 11:09:00.680085 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.680085, event: started, request: pg_notify(16.feb(1) epoch 10842) v5
  272. -98> 2014-06-02 11:09:00.680095 7f98392d2700 5 -- op tracker -- , seq: 804726, time: 2014-06-02 11:09:00.680095, event: done, request: pg_notify(16.feb(1) epoch 10842) v5
  273. -97> 2014-06-02 11:09:00.680104 7f98392d2700 10 osd.118 10842 do_waiters -- start
  274. -96> 2014-06-02 11:09:00.680107 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  275. -95> 2014-06-02 11:09:00.680111 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.17 192.168.1.101:6808/86030979 13 ==== pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (3634432978 0 0) 0x111c5a00 con 0xaf1a100
  276. -94> 2014-06-02 11:09:00.680126 7f98392d2700 10 osd.118 10842 do_waiters -- start
  277. -93> 2014-06-02 11:09:00.680128 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  278. -92> 2014-06-02 11:09:00.680129 7f98392d2700 20 osd.118 10842 _dispatch 0x111c5a00 pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  279. -91> 2014-06-02 11:09:00.680132 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.578167, event: header_read, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  280. -90> 2014-06-02 11:09:00.680137 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.578168, event: throttled, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  281. -89> 2014-06-02 11:09:00.680141 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.578229, event: all_read, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  282. -88> 2014-06-02 11:09:00.680146 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.680126, event: dispatched, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  283. -87> 2014-06-02 11:09:00.680153 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.680152, event: waiting_for_osdmap, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  284. -86> 2014-06-02 11:09:00.680159 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.17
  285. -85> 2014-06-02 11:09:00.680165 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x111c5a00
  286. -84> 2014-06-02 11:09:00.680167 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.680167, event: started, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  287. -83> 2014-06-02 11:09:00.680176 7f98392d2700 5 -- op tracker -- , seq: 804727, time: 2014-06-02 11:09:00.680176, event: done, request: pg_log(16.e38 epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  288. -82> 2014-06-02 11:09:00.680186 7f98392d2700 10 osd.118 10842 do_waiters -- start
  289. -81> 2014-06-02 11:09:00.680189 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  290. -80> 2014-06-02 11:09:00.680194 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.18 192.168.1.104:6828/94060687 17 ==== pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 ==== 762+0+0 (594420750 0 0) 0x56c4e00 con 0xaf1a3c0
  291. -79> 2014-06-02 11:09:00.680202 7f98392d2700 10 osd.118 10842 do_waiters -- start
  292. -78> 2014-06-02 11:09:00.680204 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  293. -77> 2014-06-02 11:09:00.680205 7f98392d2700 20 osd.118 10842 _dispatch 0x56c4e00 pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  294. -76> 2014-06-02 11:09:00.680209 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.478418, event: header_read, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  295. -75> 2014-06-02 11:09:00.680214 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.478420, event: throttled, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  296. -74> 2014-06-02 11:09:00.680218 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.478510, event: all_read, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  297. -73> 2014-06-02 11:09:00.680222 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.680202, event: dispatched, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  298. -72> 2014-06-02 11:09:00.680229 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.680229, event: waiting_for_osdmap, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  299. -71> 2014-06-02 11:09:00.680236 7f98392d2700 7 osd.118 10842 handle_pg_log pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4 from osd.18
  300. -70> 2014-06-02 11:09:00.680241 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x56c4e00
  301. -69> 2014-06-02 11:09:00.680244 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.680244, event: started, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  302. -68> 2014-06-02 11:09:00.680253 7f98392d2700 5 -- op tracker -- , seq: 804728, time: 2014-06-02 11:09:00.680253, event: done, request: pg_log(14.27a epoch 10842 log log((0'0,0'0], crt=0'0) query_epoch 10842) v4
  303. -67> 2014-06-02 11:09:00.680262 7f98392d2700 10 osd.118 10842 do_waiters -- start
  304. -66> 2014-06-02 11:09:00.680264 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  305. -65> 2014-06-02 11:09:00.680267 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.33 192.168.1.103:6822/77017426 15 ==== pg_notify(6.281(1) epoch 10842) v5 ==== 769+0+0 (2443130291 0 0) 0xd697380 con 0x57a7220
  306. -64> 2014-06-02 11:09:00.680274 7f98392d2700 10 osd.118 10842 do_waiters -- start
  307. -63> 2014-06-02 11:09:00.680275 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  308. -62> 2014-06-02 11:09:00.680276 7f98392d2700 20 osd.118 10842 _dispatch 0xd697380 pg_notify(6.281(1) epoch 10842) v5
  309. -61> 2014-06-02 11:09:00.680279 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.417625, event: header_read, request: pg_notify(6.281(1) epoch 10842) v5
  310. -60> 2014-06-02 11:09:00.680282 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.417628, event: throttled, request: pg_notify(6.281(1) epoch 10842) v5
  311. -59> 2014-06-02 11:09:00.680286 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.417716, event: all_read, request: pg_notify(6.281(1) epoch 10842) v5
  312. -58> 2014-06-02 11:09:00.680289 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.680274, event: dispatched, request: pg_notify(6.281(1) epoch 10842) v5
  313. -57> 2014-06-02 11:09:00.680292 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.680292, event: waiting_for_osdmap, request: pg_notify(6.281(1) epoch 10842) v5
  314. -56> 2014-06-02 11:09:00.680296 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.33
  315. -55> 2014-06-02 11:09:00.680297 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0xd697380
  316. -54> 2014-06-02 11:09:00.680299 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.680299, event: started, request: pg_notify(6.281(1) epoch 10842) v5
  317. -53> 2014-06-02 11:09:00.680311 7f98392d2700 5 -- op tracker -- , seq: 804729, time: 2014-06-02 11:09:00.680310, event: done, request: pg_notify(6.281(1) epoch 10842) v5
  318. -52> 2014-06-02 11:09:00.680319 7f98392d2700 10 osd.118 10842 do_waiters -- start
  319. -51> 2014-06-02 11:09:00.680323 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  320. -50> 2014-06-02 11:09:00.680327 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.130 192.168.1.115:6809/11028428 13 ==== pg_info(1 pgs e10842:3.191(1)) v4 ==== 765+0+0 (560351203 0 0) 0x1d4939c0 con 0x3a569e0
  321. -49> 2014-06-02 11:09:00.680336 7f98392d2700 10 osd.118 10842 do_waiters -- start
  322. -48> 2014-06-02 11:09:00.680337 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  323. -47> 2014-06-02 11:09:00.680338 7f98392d2700 20 osd.118 10842 _dispatch 0x1d4939c0 pg_info(1 pgs e10842:3.191(1)) v4
  324. -46> 2014-06-02 11:09:00.680341 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.589338, event: header_read, request: pg_info(1 pgs e10842:3.191(1)) v4
  325. -45> 2014-06-02 11:09:00.680345 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.589339, event: throttled, request: pg_info(1 pgs e10842:3.191(1)) v4
  326. -44> 2014-06-02 11:09:00.680348 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.589395, event: all_read, request: pg_info(1 pgs e10842:3.191(1)) v4
  327. -43> 2014-06-02 11:09:00.680353 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.680336, event: dispatched, request: pg_info(1 pgs e10842:3.191(1)) v4
  328. -42> 2014-06-02 11:09:00.680359 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.680359, event: waiting_for_osdmap, request: pg_info(1 pgs e10842:3.191(1)) v4
  329. -41> 2014-06-02 11:09:00.680365 7f98392d2700 7 osd.118 10842 handle_pg_info pg_info(1 pgs e10842:3.191(1)) v4 from osd.130
  330. -40> 2014-06-02 11:09:00.680370 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x1d4939c0
  331. -39> 2014-06-02 11:09:00.680372 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.680372, event: started, request: pg_info(1 pgs e10842:3.191(1)) v4
  332. -38> 2014-06-02 11:09:00.680384 7f98392d2700 5 -- op tracker -- , seq: 804730, time: 2014-06-02 11:09:00.680384, event: done, request: pg_info(1 pgs e10842:3.191(1)) v4
  333. -37> 2014-06-02 11:09:00.680393 7f98392d2700 10 osd.118 10842 do_waiters -- start
  334. -36> 2014-06-02 11:09:00.680396 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  335. -35> 2014-06-02 11:09:00.680401 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.41 192.168.1.101:6845/14054612 20 ==== pg_notify(16.927(463) epoch 10842) v5 ==== 24611+0+0 (3521848940 0 0) 0x14800380 con 0x57a32e0
  336. -34> 2014-06-02 11:09:00.680409 7f98392d2700 10 osd.118 10842 do_waiters -- start
  337. -33> 2014-06-02 11:09:00.680411 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  338. -32> 2014-06-02 11:09:00.680412 7f98392d2700 20 osd.118 10842 _dispatch 0x14800380 pg_notify(16.927(463) epoch 10842) v5
  339. -31> 2014-06-02 11:09:00.680414 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.611955, event: header_read, request: pg_notify(16.927(463) epoch 10842) v5
  340. -30> 2014-06-02 11:09:00.680418 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.611957, event: throttled, request: pg_notify(16.927(463) epoch 10842) v5
  341. -29> 2014-06-02 11:09:00.680421 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.612442, event: all_read, request: pg_notify(16.927(463) epoch 10842) v5
  342. -28> 2014-06-02 11:09:00.680424 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.680409, event: dispatched, request: pg_notify(16.927(463) epoch 10842) v5
  343. -27> 2014-06-02 11:09:00.680429 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.680429, event: waiting_for_osdmap, request: pg_notify(16.927(463) epoch 10842) v5
  344. -26> 2014-06-02 11:09:00.680435 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.41
  345. -25> 2014-06-02 11:09:00.680437 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x14800380
  346. -24> 2014-06-02 11:09:00.680439 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.680439, event: started, request: pg_notify(16.927(463) epoch 10842) v5
  347. -23> 2014-06-02 11:09:00.680450 7f98392d2700 5 -- op tracker -- , seq: 804731, time: 2014-06-02 11:09:00.680450, event: done, request: pg_notify(16.927(463) epoch 10842) v5
  348. -22> 2014-06-02 11:09:00.680537 7f98392d2700 10 osd.118 10842 do_waiters -- start
  349. -21> 2014-06-02 11:09:00.680543 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  350. -20> 2014-06-02 11:09:00.680629 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.121 192.168.1.112:6801/12006135 16 ==== pg_notify(1.173(557) epoch 10842) v5 ==== 30445+0+0 (1497930911 0 0) 0x1c03af40 con 0x1732c360
  351. -19> 2014-06-02 11:09:00.680642 7f98392d2700 10 osd.118 10842 do_waiters -- start
  352. -18> 2014-06-02 11:09:00.680644 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  353. -17> 2014-06-02 11:09:00.680645 7f98392d2700 20 osd.118 10842 _dispatch 0x1c03af40 pg_notify(1.173(557) epoch 10842) v5
  354. -16> 2014-06-02 11:09:00.680648 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.525959, event: header_read, request: pg_notify(1.173(557) epoch 10842) v5
  355. -15> 2014-06-02 11:09:00.680654 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.525961, event: throttled, request: pg_notify(1.173(557) epoch 10842) v5
  356. -14> 2014-06-02 11:09:00.680657 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.527460, event: all_read, request: pg_notify(1.173(557) epoch 10842) v5
  357. -13> 2014-06-02 11:09:00.680660 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.680642, event: dispatched, request: pg_notify(1.173(557) epoch 10842) v5
  358. -12> 2014-06-02 11:09:00.680664 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.680664, event: waiting_for_osdmap, request: pg_notify(1.173(557) epoch 10842) v5
  359. -11> 2014-06-02 11:09:00.680670 7f98392d2700 7 osd.118 10842 handle_pg_notify from osd.121
  360. -10> 2014-06-02 11:09:00.680671 7f98392d2700 15 osd.118 10842 require_same_or_newer_map 10842 (i am 10842) 0x1c03af40
  361. -9> 2014-06-02 11:09:00.680674 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.680674, event: started, request: pg_notify(1.173(557) epoch 10842) v5
  362. -8> 2014-06-02 11:09:00.680686 7f98392d2700 5 -- op tracker -- , seq: 804732, time: 2014-06-02 11:09:00.680686, event: done, request: pg_notify(1.173(557) epoch 10842) v5
  363. -7> 2014-06-02 11:09:00.680695 7f98392d2700 10 osd.118 10842 do_waiters -- start
  364. -6> 2014-06-02 11:09:00.680698 7f98392d2700 10 osd.118 10842 do_waiters -- finish
  365. -5> 2014-06-02 11:09:00.680766 7f9846d1b700 5 osd.118 10842 tick
  366. -4> 2014-06-02 11:09:00.680780 7f98392d2700 1 -- 192.168.1.114:6816/90018189 <== osd.64 192.168.1.110:6803/93028806 12 ==== pg_info(1 pgs e10842:16.21) v4 ==== 714+0+0 (3039660974 0 0) 0x78e0700 con 0xb03eb40
  367. -3> 2014-06-02 11:09:00.706943 7f97e6884700 1 -- 192.168.100.114:6844/90018189 >> :/0 pipe(0x5b7dc80 sd=722 :6844 s=0 pgs=0 cs=0 l=0 c=0x72106e0).accept sd=722 192.168.100.102:49546/0
  368. -2> 2014-06-02 11:09:00.723892 7f97e6682700 1 -- 192.168.1.114:6854/90018189 >> :/0 pipe(0xfaa2d00 sd=723 :6854 s=0 pgs=0 cs=0 l=0 c=0x1164b5a0).accept sd=723 192.168.1.102:40320/0
  369. -1> 2014-06-02 11:09:00.734831 7f97e6480700 1 -- 192.168.1.114:6854/90018189 >> :/0 pipe(0xfaa0c80 sd=724 :6854 s=0 pgs=0 cs=0 l=0 c=0x11648580).accept sd=724 192.168.1.111:38153/0
  370. 0> 2014-06-02 11:09:00.734897 7f97f425d700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f97f425d700 time 2014-06-02 11:09:00.549538
  371. msg/Pipe.cc: 1070: FAILED assert(m)
  372.  
  373. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  374. 1: (Pipe::connect()+0x3d0c) [0xb2931c]
  375. 2: (Pipe::writer()+0x9f3) [0xb2a063]
  376. 3: (Pipe::Writer::entry()+0xd) [0xb359cd]
  377. 4: (()+0x7851) [0x7f984d062851]
  378. 5: (clone()+0x6d) [0x7f984bd9e94d]
  379. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  380.  
  381. --- logging levels ---
  382. 0/ 5 none
  383. 0/ 1 lockdep
  384. 0/ 1 context
  385. 1/ 1 crush
  386. 1/ 5 mds
  387. 1/ 5 mds_balancer
  388. 1/ 5 mds_locker
  389. 1/ 5 mds_log
  390. 1/ 5 mds_log_expire
  391. 1/ 5 mds_migrator
  392. 0/ 1 buffer
  393. 0/ 1 timer
  394. 0/ 1 filer
  395. 0/ 1 striper
  396. 0/ 1 objecter
  397. 0/ 5 rados
  398. 0/ 5 rbd
  399. 0/ 5 journaler
  400. 0/ 5 objectcacher
  401. 0/ 5 client
  402. 20/20 osd
  403. 0/ 5 optracker
  404. 0/ 5 objclass
  405. 1/ 3 filestore
  406. 1/ 3 keyvaluestore
  407. 1/ 3 journal
  408. 0/ 5 ms
  409. 1/ 5 mon
  410. 0/10 monc
  411. 1/ 5 paxos
  412. 0/ 5 tp
  413. 1/ 5 auth
  414. 1/ 5 crypto
  415. 1/ 1 finisher
  416. 1/ 5 heartbeatmap
  417. 1/ 5 perfcounter
  418. 1/ 5 rgw
  419. 1/ 5 javaclient
  420. 1/ 5 asok
  421. 1/ 1 throttle
  422. -2/-2 (syslog threshold)
  423. -1/-1 (stderr threshold)
  424. max_recent 10000
  425. max_new 1000
  426. log_file /ceph/storage0114/log/ceph/ceph-osd.118.log
  427. --- end dump of recent events ---
  428. 2014-06-02 11:09:00.734907 7f97df30f700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f97df30f700 time 2014-06-02 11:09:00.549456
  429. msg/Pipe.cc: 1070: FAILED assert(m)
  430.  
  431. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  432. 1: (Pipe::connect()+0x3d0c) [0xb2931c]
  433. 2: (Pipe::writer()+0x9f3) [0xb2a063]
  434. 3: (Pipe::Writer::entry()+0xd) [0xb359cd]
  435. 4: (()+0x7851) [0x7f984d062851]
  436. 5: (clone()+0x6d) [0x7f984bd9e94d]
  437. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  438.  
  439. --- begin dump of recent events ---
  440. -3> 2014-06-02 11:09:00.734907 7f97df30f700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f97df30f700 time 2014-06-02 11:09:00.549456
  441. msg/Pipe.cc: 1070: FAILED assert(m)
  442.  
  443. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  444. 1: (Pipe::connect()+0x3d0c) [0xb2931c]
  445. 2: (Pipe::writer()+0x9f3) [0xb2a063]
  446. 3: (Pipe::Writer::entry()+0xd) [0xb359cd]
  447. 4: (()+0x7851) [0x7f984d062851]
  448. 5: (clone()+0x6d) [0x7f984bd9e94d]
  449. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  450.  
  451. -2> 2014-06-02 11:09:00.734913 7f97e637f700 1 -- 192.168.100.114:6844/90018189 >> :/0 pipe(0x5b7bc00 sd=725 :6844 s=0 pgs=0 cs=0 l=0 c=0x7214ba0).accept sd=725 192.168.100.111:44273/0
  452. -1> 2014-06-02 11:09:00.763997 7f97e607c700 1 -- 192.168.1.114:6854/90018189 >> :/0 pipe(0xfaa3c00 sd=726 :6854 s=0 pgs=0 cs=0 l=0 c=0x57a3f40).accept sd=726 192.168.1.113:43814/0
  453. 0> 2014-06-02 11:09:00.764057 7f97e5f7b700 1 -- 192.168.100.114:6844/90018189 >> :/0 pipe(0x5b78280 sd=727 :6844 s=0 pgs=0 cs=0 l=0 c=0x7215d80).accept sd=727 192.168.100.113:52859/0
  454. --- logging levels ---
  455. 0/ 5 none
  456. 0/ 1 lockdep
  457. 0/ 1 context
  458. 1/ 1 crush
  459. 1/ 5 mds
  460. 1/ 5 mds_balancer
  461. 1/ 5 mds_locker
  462. 1/ 5 mds_log
  463. 1/ 5 mds_log_expire
  464. 1/ 5 mds_migrator
  465. 0/ 1 buffer
  466. 0/ 1 timer
  467. 0/ 1 filer
  468. 0/ 1 striper
  469. 0/ 1 objecter
  470. 0/ 5 rados
  471. 0/ 5 rbd
  472. 0/ 5 journaler
  473. 0/ 5 objectcacher
  474. 0/ 5 client
  475. 20/20 osd
  476. 0/ 5 optracker
  477. 0/ 5 objclass
  478. 1/ 3 filestore
  479. 1/ 3 keyvaluestore
  480. 1/ 3 journal
  481. 0/ 5 ms
  482. 1/ 5 mon
  483. 0/10 monc
  484. 1/ 5 paxos
  485. 0/ 5 tp
  486. 1/ 5 auth
  487. 1/ 5 crypto
  488. 1/ 1 finisher
  489. 1/ 5 heartbeatmap
  490. 1/ 5 perfcounter
  491. 1/ 5 rgw
  492. 1/ 5 javaclient
  493. 1/ 5 asok
  494. 1/ 1 throttle
  495. -2/-2 (syslog threshold)
  496. -1/-1 (stderr threshold)
  497. max_recent 10000
  498. max_new 1000
  499. log_file /ceph/storage0114/log/ceph/ceph-osd.118.log
  500. --- end dump of recent events ---
  501. [root@storage0114-ib ceph]#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!