API tools faq
paste
Advertisement
alohamora

osd.123

alohamora
Jun 2nd, 2014
292
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 120.11 KB | None | 0 0
raw download clone embed print report
  1. [root@storage0114-ib ceph]# tail -1000 ceph-osd.123.log
  2. -854> 2014-06-02 11:16:18.253423 7f02026f0700 10 osd.123 11123 do_waiters -- start
  3. -853> 2014-06-02 11:16:18.253425 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  4. -852> 2014-06-02 11:16:18.253426 7f02026f0700 20 osd.123 11123 _dispatch 0x16d018c0 osd_map(11115..11115 src has 3254..11115) v3
  5. -851> 2014-06-02 11:16:18.253430 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11115,11115], i have 11123, src has [3254,11115]
  6. -850> 2014-06-02 11:16:18.253433 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  7. -849> 2014-06-02 11:16:18.253434 7f02026f0700 10 osd.123 11123 do_waiters -- start
  8. -848> 2014-06-02 11:16:18.253435 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  9. -847> 2014-06-02 11:16:18.253439 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.155 192.168.1.111:6801/78033857 99 ==== pg_notify(9.3ff(441) epoch 11116) v5 ==== 22777+0+0 (2372128223 0 0) 0x19a04ec0 con 0x17ca5540
  10. -846> 2014-06-02 11:16:18.253447 7f02026f0700 10 osd.123 11123 do_waiters -- start
  11. -845> 2014-06-02 11:16:18.253448 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  12. -844> 2014-06-02 11:16:18.253449 7f02026f0700 20 osd.123 11123 _dispatch 0x19a04ec0 pg_notify(9.3ff(441) epoch 11116) v5
  13. -843> 2014-06-02 11:16:18.253464 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:06.926042, event: header_read, request: pg_notify(9.3ff(441) epoch 11116) v5
  14. -842> 2014-06-02 11:16:18.253468 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:06.926044, event: throttled, request: pg_notify(9.3ff(441) epoch 11116) v5
  15. -841> 2014-06-02 11:16:18.253471 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:06.926682, event: all_read, request: pg_notify(9.3ff(441) epoch 11116) v5
  16. -840> 2014-06-02 11:16:18.253475 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:18.253447, event: dispatched, request: pg_notify(9.3ff(441) epoch 11116) v5
  17. -839> 2014-06-02 11:16:18.253479 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:18.253478, event: waiting_for_osdmap, request: pg_notify(9.3ff(441) epoch 11116) v5
  18. -838> 2014-06-02 11:16:18.253482 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.155
  19. -837> 2014-06-02 11:16:18.253484 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x19a04ec0
  20. -836> 2014-06-02 11:16:18.253486 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(9.3ff(441) epoch 11116) v5
  21. -835> 2014-06-02 11:16:18.253489 7f02026f0700 5 -- op tracker -- , seq: 150774, time: 2014-06-02 11:16:18.253489, event: done, request: pg_notify(9.3ff(441) epoch 11116) v5
  22. -834> 2014-06-02 11:16:18.253496 7f02026f0700 10 osd.123 11123 do_waiters -- start
  23. -833> 2014-06-02 11:16:18.253497 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  24. -832> 2014-06-02 11:16:18.253582 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.150 192.168.1.111:6818/102010870 40 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x5bb2640 con 0x1a3769e0
  25. -831> 2014-06-02 11:16:18.253605 7f02026f0700 10 osd.123 11123 do_waiters -- start
  26. -830> 2014-06-02 11:16:18.253607 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  27. -829> 2014-06-02 11:16:18.253608 7f02026f0700 20 osd.123 11123 _dispatch 0x5bb2640 osd_map(11116..11116 src has 3254..11116) v3
  28. -828> 2014-06-02 11:16:18.253612 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  29. -827> 2014-06-02 11:16:18.253617 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  30. -826> 2014-06-02 11:16:18.253619 7f02026f0700 10 osd.123 11123 do_waiters -- start
  31. -825> 2014-06-02 11:16:18.253620 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  32. -824> 2014-06-02 11:16:18.253624 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 66 ==== osd_map(11115..11115 src has 3254..11115) v3 ==== 1715+0+0 (4177096548 0 0) 0x6631440 con 0x10e32100
  33. -823> 2014-06-02 11:16:18.253632 7f02026f0700 10 osd.123 11123 do_waiters -- start
  34. -822> 2014-06-02 11:16:18.253633 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  35. -821> 2014-06-02 11:16:18.253634 7f02026f0700 20 osd.123 11123 _dispatch 0x6631440 osd_map(11115..11115 src has 3254..11115) v3
  36. -820> 2014-06-02 11:16:18.253638 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11115,11115], i have 11123, src has [3254,11115]
  37. -819> 2014-06-02 11:16:18.253641 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  38. -818> 2014-06-02 11:16:18.253642 7f02026f0700 10 osd.123 11123 do_waiters -- start
  39. -817> 2014-06-02 11:16:18.253643 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  40. -816> 2014-06-02 11:16:18.253647 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 65 ==== pg_notify(8.127(13) epoch 11114) v5 ==== 1397+0+0 (3055403888 0 0) 0x90db9c0 con 0x10589a20
  41. -815> 2014-06-02 11:16:18.253655 7f02026f0700 10 osd.123 11123 do_waiters -- start
  42. -814> 2014-06-02 11:16:18.253656 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  43. -813> 2014-06-02 11:16:18.253657 7f02026f0700 20 osd.123 11123 _dispatch 0x90db9c0 pg_notify(8.127(13) epoch 11114) v5
  44. -812> 2014-06-02 11:16:18.253661 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:04.694954, event: header_read, request: pg_notify(8.127(13) epoch 11114) v5
  45. -811> 2014-06-02 11:16:18.253665 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:04.694956, event: throttled, request: pg_notify(8.127(13) epoch 11114) v5
  46. -810> 2014-06-02 11:16:18.253669 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:04.695084, event: all_read, request: pg_notify(8.127(13) epoch 11114) v5
  47. -809> 2014-06-02 11:16:18.253672 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:18.253655, event: dispatched, request: pg_notify(8.127(13) epoch 11114) v5
  48. -808> 2014-06-02 11:16:18.253676 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:18.253675, event: waiting_for_osdmap, request: pg_notify(8.127(13) epoch 11114) v5
  49. -807> 2014-06-02 11:16:18.253680 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  50. -806> 2014-06-02 11:16:18.253682 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0x90db9c0
  51. -805> 2014-06-02 11:16:18.253684 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(8.127(13) epoch 11114) v5
  52. -804> 2014-06-02 11:16:18.253688 7f02026f0700 5 -- op tracker -- , seq: 150775, time: 2014-06-02 11:16:18.253688, event: done, request: pg_notify(8.127(13) epoch 11114) v5
  53. -803> 2014-06-02 11:16:18.253706 7f02026f0700 10 osd.123 11123 do_waiters -- start
  54. -802> 2014-06-02 11:16:18.253707 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  55. -801> 2014-06-02 11:16:18.253712 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.22 192.168.1.101:6810/109034609 38 ==== pg_notify(16.dc8(713) epoch 11115) v5 ==== 38273+0+0 (3443193731 0 0) 0x193a8c40 con 0x10590c60
  56. -800> 2014-06-02 11:16:18.253719 7f02026f0700 10 osd.123 11123 do_waiters -- start
  57. -799> 2014-06-02 11:16:18.253720 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  58. -798> 2014-06-02 11:16:18.253722 7f02026f0700 20 osd.123 11123 _dispatch 0x193a8c40 pg_notify(16.dc8(713) epoch 11115) v5
  59. -797> 2014-06-02 11:16:18.253724 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:05.985157, event: header_read, request: pg_notify(16.dc8(713) epoch 11115) v5
  60. -796> 2014-06-02 11:16:18.253728 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:05.985160, event: throttled, request: pg_notify(16.dc8(713) epoch 11115) v5
  61. -795> 2014-06-02 11:16:18.253732 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:05.986174, event: all_read, request: pg_notify(16.dc8(713) epoch 11115) v5
  62. -794> 2014-06-02 11:16:18.253735 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:18.253719, event: dispatched, request: pg_notify(16.dc8(713) epoch 11115) v5
  63. -793> 2014-06-02 11:16:18.253739 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:18.253738, event: waiting_for_osdmap, request: pg_notify(16.dc8(713) epoch 11115) v5
  64. -792> 2014-06-02 11:16:18.253742 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.22
  65. -791> 2014-06-02 11:16:18.253744 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x193a8c40
  66. -790> 2014-06-02 11:16:18.253745 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.dc8(713) epoch 11115) v5
  67. -789> 2014-06-02 11:16:18.253748 7f02026f0700 5 -- op tracker -- , seq: 150776, time: 2014-06-02 11:16:18.253748, event: done, request: pg_notify(16.dc8(713) epoch 11115) v5
  68. -788> 2014-06-02 11:16:18.253754 7f02026f0700 10 osd.123 11123 do_waiters -- start
  69. -787> 2014-06-02 11:16:18.253768 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  70. -786> 2014-06-02 11:16:18.253855 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 40 ==== pg_notify(9.2fe(550) epoch 11114) v5 ==== 28872+0+0 (760753976 0 0) 0xe630a80 con 0x10590000
  71. -785> 2014-06-02 11:16:18.253868 7f02026f0700 10 osd.123 11123 do_waiters -- start
  72. -784> 2014-06-02 11:16:18.253870 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  73. -783> 2014-06-02 11:16:18.253871 7f02026f0700 20 osd.123 11123 _dispatch 0xe630a80 pg_notify(9.2fe(550) epoch 11114) v5
  74. -782> 2014-06-02 11:16:18.253875 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:04.830153, event: header_read, request: pg_notify(9.2fe(550) epoch 11114) v5
  75. -781> 2014-06-02 11:16:18.253879 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:04.830155, event: throttled, request: pg_notify(9.2fe(550) epoch 11114) v5
  76. -780> 2014-06-02 11:16:18.253883 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:04.831022, event: all_read, request: pg_notify(9.2fe(550) epoch 11114) v5
  77. -779> 2014-06-02 11:16:18.253886 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:18.253867, event: dispatched, request: pg_notify(9.2fe(550) epoch 11114) v5
  78. -778> 2014-06-02 11:16:18.253890 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:18.253890, event: waiting_for_osdmap, request: pg_notify(9.2fe(550) epoch 11114) v5
  79. -777> 2014-06-02 11:16:18.253895 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  80. -776> 2014-06-02 11:16:18.253899 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0xe630a80
  81. -775> 2014-06-02 11:16:18.253901 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(9.2fe(550) epoch 11114) v5
  82. -774> 2014-06-02 11:16:18.253904 7f02026f0700 5 -- op tracker -- , seq: 150777, time: 2014-06-02 11:16:18.253904, event: done, request: pg_notify(9.2fe(550) epoch 11114) v5
  83. -773> 2014-06-02 11:16:18.253911 7f02026f0700 10 osd.123 11123 do_waiters -- start
  84. -772> 2014-06-02 11:16:18.253913 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  85. -771> 2014-06-02 11:16:18.253993 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.33 192.168.1.103:6809/83017426 32 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x6632f40 con 0x10eb5800
  86. -770> 2014-06-02 11:16:18.254004 7f02026f0700 10 osd.123 11123 do_waiters -- start
  87. -769> 2014-06-02 11:16:18.254006 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  88. -768> 2014-06-02 11:16:18.254008 7f02026f0700 20 osd.123 11123 _dispatch 0x6632f40 osd_map(11116..11116 src has 3254..11116) v3
  89. -767> 2014-06-02 11:16:18.254012 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  90. -766> 2014-06-02 11:16:18.254015 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  91. -765> 2014-06-02 11:16:18.254016 7f02026f0700 10 osd.123 11123 do_waiters -- start
  92. -764> 2014-06-02 11:16:18.254029 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  93. -763> 2014-06-02 11:16:18.254034 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 25 ==== pg_notify(13.330(17) epoch 11113) v5 ==== 1633+0+0 (865588941 0 0) 0x18dc4980 con 0x10c75c20
  94. -762> 2014-06-02 11:16:18.254042 7f02026f0700 10 osd.123 11123 do_waiters -- start
  95. -761> 2014-06-02 11:16:18.254043 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  96. -760> 2014-06-02 11:16:18.254044 7f02026f0700 20 osd.123 11123 _dispatch 0x18dc4980 pg_notify(13.330(17) epoch 11113) v5
  97. -759> 2014-06-02 11:16:18.254047 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:03.658823, event: header_read, request: pg_notify(13.330(17) epoch 11113) v5
  98. -758> 2014-06-02 11:16:18.254052 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:03.658823, event: throttled, request: pg_notify(13.330(17) epoch 11113) v5
  99. -757> 2014-06-02 11:16:18.254056 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:03.658884, event: all_read, request: pg_notify(13.330(17) epoch 11113) v5
  100. -756> 2014-06-02 11:16:18.254059 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:18.254041, event: dispatched, request: pg_notify(13.330(17) epoch 11113) v5
  101. -755> 2014-06-02 11:16:18.254063 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:18.254063, event: waiting_for_osdmap, request: pg_notify(13.330(17) epoch 11113) v5
  102. -754> 2014-06-02 11:16:18.254067 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  103. -753> 2014-06-02 11:16:18.254068 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11113 (i am 11123) 0x18dc4980
  104. -752> 2014-06-02 11:16:18.254071 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(13.330(17) epoch 11113) v5
  105. -751> 2014-06-02 11:16:18.254074 7f02026f0700 5 -- op tracker -- , seq: 150778, time: 2014-06-02 11:16:18.254074, event: done, request: pg_notify(13.330(17) epoch 11113) v5
  106. -750> 2014-06-02 11:16:18.254081 7f02026f0700 10 osd.123 11123 do_waiters -- start
  107. -749> 2014-06-02 11:16:18.254083 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  108. -748> 2014-06-02 11:16:18.254088 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.70 192.168.1.109:6811/18033379 19 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x16d06e40 con 0x10c71080
  109. -747> 2014-06-02 11:16:18.254096 7f02026f0700 10 osd.123 11123 do_waiters -- start
  110. -746> 2014-06-02 11:16:18.254100 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  111. -745> 2014-06-02 11:16:18.254102 7f02026f0700 20 osd.123 11123 _dispatch 0x16d06e40 osd_map(11116..11116 src has 3254..11116) v3
  112. -744> 2014-06-02 11:16:18.254106 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  113. -743> 2014-06-02 11:16:18.254109 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  114. -742> 2014-06-02 11:16:18.254110 7f02026f0700 10 osd.123 11123 do_waiters -- start
  115. -741> 2014-06-02 11:16:18.254111 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  116. -740> 2014-06-02 11:16:18.254115 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.76 192.168.1.106:6807/95037986 104 ==== pg_notify(15.279(19) epoch 11115) v5 ==== 1735+0+0 (2225257208 0 0) 0xe633100 con 0x2e87a60
  117. -739> 2014-06-02 11:16:18.254123 7f02026f0700 10 osd.123 11123 do_waiters -- start
  118. -738> 2014-06-02 11:16:18.254124 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  119. -737> 2014-06-02 11:16:18.254125 7f02026f0700 20 osd.123 11123 _dispatch 0xe633100 pg_notify(15.279(19) epoch 11115) v5
  120. -736> 2014-06-02 11:16:18.254140 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:06.055289, event: header_read, request: pg_notify(15.279(19) epoch 11115) v5
  121. -735> 2014-06-02 11:16:18.254143 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:06.055291, event: throttled, request: pg_notify(15.279(19) epoch 11115) v5
  122. -734> 2014-06-02 11:16:18.254147 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:06.055449, event: all_read, request: pg_notify(15.279(19) epoch 11115) v5
  123. -733> 2014-06-02 11:16:18.254150 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:18.254122, event: dispatched, request: pg_notify(15.279(19) epoch 11115) v5
  124. -732> 2014-06-02 11:16:18.254158 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:18.254158, event: waiting_for_osdmap, request: pg_notify(15.279(19) epoch 11115) v5
  125. -731> 2014-06-02 11:16:18.254162 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.76
  126. -730> 2014-06-02 11:16:18.254164 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xe633100
  127. -729> 2014-06-02 11:16:18.254166 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.279(19) epoch 11115) v5
  128. -728> 2014-06-02 11:16:18.254168 7f02026f0700 5 -- op tracker -- , seq: 150779, time: 2014-06-02 11:16:18.254168, event: done, request: pg_notify(15.279(19) epoch 11115) v5
  129. -727> 2014-06-02 11:16:18.254174 7f02026f0700 10 osd.123 11123 do_waiters -- start
  130. -726> 2014-06-02 11:16:18.254176 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  131. -725> 2014-06-02 11:16:18.254181 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.150 192.168.1.111:6818/102010870 41 ==== pg_notify(7.293(15) epoch 11116) v5 ==== 1531+0+0 (111624905 0 0) 0x1804ec80 con 0x1a3769e0
  132. -724> 2014-06-02 11:16:18.254189 7f02026f0700 10 osd.123 11123 do_waiters -- start
  133. -723> 2014-06-02 11:16:18.254202 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  134. -722> 2014-06-02 11:16:18.254203 7f02026f0700 20 osd.123 11123 _dispatch 0x1804ec80 pg_notify(7.293(15) epoch 11116) v5
  135. -721> 2014-06-02 11:16:18.254206 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:06.956669, event: header_read, request: pg_notify(7.293(15) epoch 11116) v5
  136. -720> 2014-06-02 11:16:18.254211 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:06.956671, event: throttled, request: pg_notify(7.293(15) epoch 11116) v5
  137. -719> 2014-06-02 11:16:18.254214 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:06.956789, event: all_read, request: pg_notify(7.293(15) epoch 11116) v5
  138. -718> 2014-06-02 11:16:18.254217 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:18.254189, event: dispatched, request: pg_notify(7.293(15) epoch 11116) v5
  139. -717> 2014-06-02 11:16:18.254221 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:18.254221, event: waiting_for_osdmap, request: pg_notify(7.293(15) epoch 11116) v5
  140. -716> 2014-06-02 11:16:18.254225 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.150
  141. -715> 2014-06-02 11:16:18.254227 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x1804ec80
  142. -714> 2014-06-02 11:16:18.254229 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(7.293(15) epoch 11116) v5
  143. -713> 2014-06-02 11:16:18.254232 7f02026f0700 5 -- op tracker -- , seq: 150780, time: 2014-06-02 11:16:18.254232, event: done, request: pg_notify(7.293(15) epoch 11116) v5
  144. -712> 2014-06-02 11:16:18.254238 7f02026f0700 10 osd.123 11123 do_waiters -- start
  145. -711> 2014-06-02 11:16:18.254239 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  146. -710> 2014-06-02 11:16:18.254245 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 67 ==== pg_notify(16.4b(14) epoch 11115) v5 ==== 1440+0+0 (1849657815 0 0) 0xa554d00 con 0x10e32100
  147. -709> 2014-06-02 11:16:18.254253 7f02026f0700 10 osd.123 11123 do_waiters -- start
  148. -708> 2014-06-02 11:16:18.254254 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  149. -707> 2014-06-02 11:16:18.254255 7f02026f0700 20 osd.123 11123 _dispatch 0xa554d00 pg_notify(16.4b(14) epoch 11115) v5
  150. -706> 2014-06-02 11:16:18.254258 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:05.929982, event: header_read, request: pg_notify(16.4b(14) epoch 11115) v5
  151. -705> 2014-06-02 11:16:18.254262 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:05.929983, event: throttled, request: pg_notify(16.4b(14) epoch 11115) v5
  152. -704> 2014-06-02 11:16:18.254266 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:05.930072, event: all_read, request: pg_notify(16.4b(14) epoch 11115) v5
  153. -703> 2014-06-02 11:16:18.254269 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:18.254252, event: dispatched, request: pg_notify(16.4b(14) epoch 11115) v5
  154. -702> 2014-06-02 11:16:18.254273 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:18.254273, event: waiting_for_osdmap, request: pg_notify(16.4b(14) epoch 11115) v5
  155. -701> 2014-06-02 11:16:18.254279 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.96
  156. -700> 2014-06-02 11:16:18.254281 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xa554d00
  157. -699> 2014-06-02 11:16:18.254283 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.4b(14) epoch 11115) v5
  158. -698> 2014-06-02 11:16:18.254286 7f02026f0700 5 -- op tracker -- , seq: 150781, time: 2014-06-02 11:16:18.254286, event: done, request: pg_notify(16.4b(14) epoch 11115) v5
  159. -697> 2014-06-02 11:16:18.254292 7f02026f0700 10 osd.123 11123 do_waiters -- start
  160. -696> 2014-06-02 11:16:18.254294 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  161. -695> 2014-06-02 11:16:18.254299 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 66 ==== pg_notify(6.1bf(9) epoch 11114) v5 ==== 1209+0+0 (1115811662 0 0) 0x90df380 con 0x10589a20
  162. -694> 2014-06-02 11:16:18.254307 7f02026f0700 10 osd.123 11123 do_waiters -- start
  163. -693> 2014-06-02 11:16:18.254310 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  164. -692> 2014-06-02 11:16:18.254312 7f02026f0700 20 osd.123 11123 _dispatch 0x90df380 pg_notify(6.1bf(9) epoch 11114) v5
  165. -691> 2014-06-02 11:16:18.254315 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:04.706535, event: header_read, request: pg_notify(6.1bf(9) epoch 11114) v5
  166. -690> 2014-06-02 11:16:18.254319 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:04.706537, event: throttled, request: pg_notify(6.1bf(9) epoch 11114) v5
  167. -689> 2014-06-02 11:16:18.254322 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:04.706663, event: all_read, request: pg_notify(6.1bf(9) epoch 11114) v5
  168. -688> 2014-06-02 11:16:18.254326 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:18.254306, event: dispatched, request: pg_notify(6.1bf(9) epoch 11114) v5
  169. -687> 2014-06-02 11:16:18.254330 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:18.254330, event: waiting_for_osdmap, request: pg_notify(6.1bf(9) epoch 11114) v5
  170. -686> 2014-06-02 11:16:18.254347 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  171. -685> 2014-06-02 11:16:18.254349 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0x90df380
  172. -684> 2014-06-02 11:16:18.254351 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1bf(9) epoch 11114) v5
  173. -683> 2014-06-02 11:16:18.254354 7f02026f0700 5 -- op tracker -- , seq: 150782, time: 2014-06-02 11:16:18.254354, event: done, request: pg_notify(6.1bf(9) epoch 11114) v5
  174. -682> 2014-06-02 11:16:18.254360 7f02026f0700 10 osd.123 11123 do_waiters -- start
  175. -681> 2014-06-02 11:16:18.254361 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  176. -680> 2014-06-02 11:16:18.254365 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.22 192.168.1.101:6810/109034609 39 ==== pg_notify(15.d6(63) epoch 11115) v5 ==== 4091+0+0 (507072580 0 0) 0x193ab800 con 0x10590c60
  177. -679> 2014-06-02 11:16:18.254373 7f02026f0700 10 osd.123 11123 do_waiters -- start
  178. -678> 2014-06-02 11:16:18.254374 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  179. -677> 2014-06-02 11:16:18.254375 7f02026f0700 20 osd.123 11123 _dispatch 0x193ab800 pg_notify(15.d6(63) epoch 11115) v5
  180. -676> 2014-06-02 11:16:18.254378 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:06.003050, event: header_read, request: pg_notify(15.d6(63) epoch 11115) v5
  181. -675> 2014-06-02 11:16:18.254382 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:06.003053, event: throttled, request: pg_notify(15.d6(63) epoch 11115) v5
  182. -674> 2014-06-02 11:16:18.254385 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:06.003244, event: all_read, request: pg_notify(15.d6(63) epoch 11115) v5
  183. -673> 2014-06-02 11:16:18.254389 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:18.254372, event: dispatched, request: pg_notify(15.d6(63) epoch 11115) v5
  184. -672> 2014-06-02 11:16:18.254393 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:18.254393, event: waiting_for_osdmap, request: pg_notify(15.d6(63) epoch 11115) v5
  185. -671> 2014-06-02 11:16:18.254409 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.22
  186. -670> 2014-06-02 11:16:18.254411 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x193ab800
  187. -669> 2014-06-02 11:16:18.254413 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.d6(63) epoch 11115) v5
  188. -668> 2014-06-02 11:16:18.254416 7f02026f0700 5 -- op tracker -- , seq: 150783, time: 2014-06-02 11:16:18.254416, event: done, request: pg_notify(15.d6(63) epoch 11115) v5
  189. -667> 2014-06-02 11:16:18.254422 7f02026f0700 10 osd.123 11123 do_waiters -- start
  190. -666> 2014-06-02 11:16:18.254424 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  191. -665> 2014-06-02 11:16:18.254434 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 41 ==== pg_notify(6.1b8(14) epoch 11114) v5 ==== 1480+0+0 (2002766124 0 0) 0xe634ec0 con 0x10590000
  192. -664> 2014-06-02 11:16:18.254442 7f02026f0700 10 osd.123 11123 do_waiters -- start
  193. -663> 2014-06-02 11:16:18.254447 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  194. -662> 2014-06-02 11:16:18.254448 7f02026f0700 20 osd.123 11123 _dispatch 0xe634ec0 pg_notify(6.1b8(14) epoch 11114) v5
  195. -661> 2014-06-02 11:16:18.254451 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:04.853696, event: header_read, request: pg_notify(6.1b8(14) epoch 11114) v5
  196. -660> 2014-06-02 11:16:18.254455 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:04.853699, event: throttled, request: pg_notify(6.1b8(14) epoch 11114) v5
  197. -659> 2014-06-02 11:16:18.254459 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:04.853822, event: all_read, request: pg_notify(6.1b8(14) epoch 11114) v5
  198. -658> 2014-06-02 11:16:18.254462 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:18.254442, event: dispatched, request: pg_notify(6.1b8(14) epoch 11114) v5
  199. -657> 2014-06-02 11:16:18.254478 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:18.254478, event: waiting_for_osdmap, request: pg_notify(6.1b8(14) epoch 11114) v5
  200. -656> 2014-06-02 11:16:18.254483 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  201. -655> 2014-06-02 11:16:18.254485 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0xe634ec0
  202. -654> 2014-06-02 11:16:18.254487 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1b8(14) epoch 11114) v5
  203. -653> 2014-06-02 11:16:18.254490 7f02026f0700 5 -- op tracker -- , seq: 150784, time: 2014-06-02 11:16:18.254490, event: done, request: pg_notify(6.1b8(14) epoch 11114) v5
  204. -652> 2014-06-02 11:16:18.254496 7f02026f0700 10 osd.123 11123 do_waiters -- start
  205. -651> 2014-06-02 11:16:18.254498 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  206. -650> 2014-06-02 11:16:18.254502 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.33 192.168.1.103:6809/83017426 33 ==== pg_notify(16.6bd(19) epoch 11116) v5 ==== 1743+0+0 (3291696203 0 0) 0xa5539c0 con 0x10eb5800
  207. -649> 2014-06-02 11:16:18.254510 7f02026f0700 10 osd.123 11123 do_waiters -- start
  208. -648> 2014-06-02 11:16:18.254511 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  209. -647> 2014-06-02 11:16:18.254512 7f02026f0700 20 osd.123 11123 _dispatch 0xa5539c0 pg_notify(16.6bd(19) epoch 11116) v5
  210. -646> 2014-06-02 11:16:18.254515 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:06.930966, event: header_read, request: pg_notify(16.6bd(19) epoch 11116) v5
  211. -645> 2014-06-02 11:16:18.254519 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:06.930967, event: throttled, request: pg_notify(16.6bd(19) epoch 11116) v5
  212. -644> 2014-06-02 11:16:18.254523 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:06.931063, event: all_read, request: pg_notify(16.6bd(19) epoch 11116) v5
  213. -643> 2014-06-02 11:16:18.254526 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:18.254509, event: dispatched, request: pg_notify(16.6bd(19) epoch 11116) v5
  214. -642> 2014-06-02 11:16:18.254543 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:18.254543, event: waiting_for_osdmap, request: pg_notify(16.6bd(19) epoch 11116) v5
  215. -641> 2014-06-02 11:16:18.254549 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.33
  216. -640> 2014-06-02 11:16:18.254552 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xa5539c0
  217. -639> 2014-06-02 11:16:18.254554 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.6bd(19) epoch 11116) v5
  218. -638> 2014-06-02 11:16:18.254557 7f02026f0700 5 -- op tracker -- , seq: 150785, time: 2014-06-02 11:16:18.254557, event: done, request: pg_notify(16.6bd(19) epoch 11116) v5
  219. -637> 2014-06-02 11:16:18.254563 7f02026f0700 10 osd.123 11123 do_waiters -- start
  220. -636> 2014-06-02 11:16:18.254565 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  221. -635> 2014-06-02 11:16:18.254570 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 26 ==== pg_notify(15.207(617) epoch 11113) v5 ==== 32585+0+0 (3272809206 0 0) 0x18dc5940 con 0x10c75c20
  222. -634> 2014-06-02 11:16:18.254578 7f02026f0700 10 osd.123 11123 do_waiters -- start
  223. -633> 2014-06-02 11:16:18.254580 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  224. -632> 2014-06-02 11:16:18.254581 7f02026f0700 20 osd.123 11123 _dispatch 0x18dc5940 pg_notify(15.207(617) epoch 11113) v5
  225. -631> 2014-06-02 11:16:18.254584 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:03.778589, event: header_read, request: pg_notify(15.207(617) epoch 11113) v5
  226. -630> 2014-06-02 11:16:18.254588 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:03.778591, event: throttled, request: pg_notify(15.207(617) epoch 11113) v5
  227. -629> 2014-06-02 11:16:18.254591 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:03.779431, event: all_read, request: pg_notify(15.207(617) epoch 11113) v5
  228. -628> 2014-06-02 11:16:18.254595 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:18.254578, event: dispatched, request: pg_notify(15.207(617) epoch 11113) v5
  229. -627> 2014-06-02 11:16:18.254612 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:18.254612, event: waiting_for_osdmap, request: pg_notify(15.207(617) epoch 11113) v5
  230. -626> 2014-06-02 11:16:18.254617 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  231. -625> 2014-06-02 11:16:18.254619 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11113 (i am 11123) 0x18dc5940
  232. -624> 2014-06-02 11:16:18.254621 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.207(617) epoch 11113) v5
  233. -623> 2014-06-02 11:16:18.254624 7f02026f0700 5 -- op tracker -- , seq: 150786, time: 2014-06-02 11:16:18.254624, event: done, request: pg_notify(15.207(617) epoch 11113) v5
  234. -622> 2014-06-02 11:16:18.254631 7f02026f0700 10 osd.123 11123 do_waiters -- start
  235. -621> 2014-06-02 11:16:18.254633 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  236. -620> 2014-06-02 11:16:18.254710 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.70 192.168.1.109:6811/18033379 20 ==== pg_notify(16.7ad(27) epoch 11116) v5 ==== 2127+0+0 (3100502435 0 0) 0x11a68e00 con 0x10c71080
  237. -619> 2014-06-02 11:16:18.254722 7f02026f0700 10 osd.123 11123 do_waiters -- start
  238. -618> 2014-06-02 11:16:18.254724 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  239. -617> 2014-06-02 11:16:18.254725 7f02026f0700 20 osd.123 11123 _dispatch 0x11a68e00 pg_notify(16.7ad(27) epoch 11116) v5
  240. -616> 2014-06-02 11:16:18.254728 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:06.899482, event: header_read, request: pg_notify(16.7ad(27) epoch 11116) v5
  241. -615> 2014-06-02 11:16:18.254733 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:06.899483, event: throttled, request: pg_notify(16.7ad(27) epoch 11116) v5
  242. -614> 2014-06-02 11:16:18.254736 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:06.899615, event: all_read, request: pg_notify(16.7ad(27) epoch 11116) v5
  243. -613> 2014-06-02 11:16:18.254739 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:18.254721, event: dispatched, request: pg_notify(16.7ad(27) epoch 11116) v5
  244. -612> 2014-06-02 11:16:18.254744 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:18.254744, event: waiting_for_osdmap, request: pg_notify(16.7ad(27) epoch 11116) v5
  245. -611> 2014-06-02 11:16:18.254750 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.70
  246. -610> 2014-06-02 11:16:18.254753 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x11a68e00
  247. -609> 2014-06-02 11:16:18.254756 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.7ad(27) epoch 11116) v5
  248. -608> 2014-06-02 11:16:18.254759 7f02026f0700 5 -- op tracker -- , seq: 150787, time: 2014-06-02 11:16:18.254758, event: done, request: pg_notify(16.7ad(27) epoch 11116) v5
  249. -607> 2014-06-02 11:16:18.254766 7f02026f0700 10 osd.123 11123 do_waiters -- start
  250. -606> 2014-06-02 11:16:18.254768 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  251. -605> 2014-06-02 11:16:18.254774 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.76 192.168.1.106:6807/95037986 105 ==== pg_notify(3.1b7(563) epoch 11115) v5 ==== 30423+0+0 (2495182587 0 0) 0xe632f40 con 0x2e87a60
  252. -604> 2014-06-02 11:16:18.254795 7f02026f0700 10 osd.123 11123 do_waiters -- start
  253. -603> 2014-06-02 11:16:18.254797 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  254. -602> 2014-06-02 11:16:18.254799 7f02026f0700 20 osd.123 11123 _dispatch 0xe632f40 pg_notify(3.1b7(563) epoch 11115) v5
  255. -601> 2014-06-02 11:16:18.254802 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:06.103981, event: header_read, request: pg_notify(3.1b7(563) epoch 11115) v5
  256. -600> 2014-06-02 11:16:18.254806 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:06.103983, event: throttled, request: pg_notify(3.1b7(563) epoch 11115) v5
  257. -599> 2014-06-02 11:16:18.254809 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:06.104969, event: all_read, request: pg_notify(3.1b7(563) epoch 11115) v5
  258. -598> 2014-06-02 11:16:18.254813 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:18.254784, event: dispatched, request: pg_notify(3.1b7(563) epoch 11115) v5
  259. -597> 2014-06-02 11:16:18.254816 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:18.254816, event: waiting_for_osdmap, request: pg_notify(3.1b7(563) epoch 11115) v5
  260. -596> 2014-06-02 11:16:18.254822 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.76
  261. -595> 2014-06-02 11:16:18.254825 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xe632f40
  262. -594> 2014-06-02 11:16:18.254827 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(563) epoch 11115) v5
  263. -593> 2014-06-02 11:16:18.254830 7f02026f0700 5 -- op tracker -- , seq: 150788, time: 2014-06-02 11:16:18.254830, event: done, request: pg_notify(3.1b7(563) epoch 11115) v5
  264. -592> 2014-06-02 11:16:18.254836 7f02026f0700 10 osd.123 11123 do_waiters -- start
  265. -591> 2014-06-02 11:16:18.254838 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  266. -590> 2014-06-02 11:16:18.254921 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 68 ==== pg_notify(5.393(523) epoch 11115) v5 ==== 26775+0+0 (3345363073 0 0) 0xa551f80 con 0x10e32100
  267. -589> 2014-06-02 11:16:18.254933 7f02026f0700 10 osd.123 11123 do_waiters -- start
  268. -588> 2014-06-02 11:16:18.254935 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  269. -587> 2014-06-02 11:16:18.254936 7f02026f0700 20 osd.123 11123 _dispatch 0xa551f80 pg_notify(5.393(523) epoch 11115) v5
  270. -586> 2014-06-02 11:16:18.254939 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:05.966266, event: header_read, request: pg_notify(5.393(523) epoch 11115) v5
  271. -585> 2014-06-02 11:16:18.254944 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:05.966268, event: throttled, request: pg_notify(5.393(523) epoch 11115) v5
  272. -584> 2014-06-02 11:16:18.254947 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:05.967025, event: all_read, request: pg_notify(5.393(523) epoch 11115) v5
  273. -583> 2014-06-02 11:16:18.254950 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:18.254933, event: dispatched, request: pg_notify(5.393(523) epoch 11115) v5
  274. -582> 2014-06-02 11:16:18.254954 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:18.254954, event: waiting_for_osdmap, request: pg_notify(5.393(523) epoch 11115) v5
  275. -581> 2014-06-02 11:16:18.254957 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.96
  276. -580> 2014-06-02 11:16:18.254959 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xa551f80
  277. -579> 2014-06-02 11:16:18.254961 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(5.393(523) epoch 11115) v5
  278. -578> 2014-06-02 11:16:18.254964 7f02026f0700 5 -- op tracker -- , seq: 150789, time: 2014-06-02 11:16:18.254964, event: done, request: pg_notify(5.393(523) epoch 11115) v5
  279. -577> 2014-06-02 11:16:18.254973 7f02026f0700 10 osd.123 11123 do_waiters -- start
  280. -576> 2014-06-02 11:16:18.254975 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  281. -575> 2014-06-02 11:16:18.255155 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 67 ==== osd_map(11115..11115 src has 3254..11115) v3 ==== 1715+0+0 (4177096548 0 0) 0x5bb3cc0 con 0x10589a20
  282. -574> 2014-06-02 11:16:18.255169 7f02026f0700 10 osd.123 11123 do_waiters -- start
  283. -573> 2014-06-02 11:16:18.255171 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  284. -572> 2014-06-02 11:16:18.255172 7f02026f0700 20 osd.123 11123 _dispatch 0x5bb3cc0 osd_map(11115..11115 src has 3254..11115) v3
  285. -571> 2014-06-02 11:16:18.255176 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11115,11115], i have 11123, src has [3254,11115]
  286. -570> 2014-06-02 11:16:18.255179 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  287. -569> 2014-06-02 11:16:18.255181 7f02026f0700 10 osd.123 11123 do_waiters -- start
  288. -568> 2014-06-02 11:16:18.255182 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  289. -567> 2014-06-02 11:16:18.255186 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.22 192.168.1.101:6810/109034609 40 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x6633840 con 0x10590c60
  290. -566> 2014-06-02 11:16:18.255193 7f02026f0700 10 osd.123 11123 do_waiters -- start
  291. -565> 2014-06-02 11:16:18.255194 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  292. -564> 2014-06-02 11:16:18.255195 7f02026f0700 20 osd.123 11123 _dispatch 0x6633840 osd_map(11116..11116 src has 3254..11116) v3
  293. -563> 2014-06-02 11:16:18.255199 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  294. -562> 2014-06-02 11:16:18.255201 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  295. -561> 2014-06-02 11:16:18.255203 7f02026f0700 10 osd.123 11123 do_waiters -- start
  296. -560> 2014-06-02 11:16:18.255204 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  297. -559> 2014-06-02 11:16:18.255207 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 42 ==== osd_map(11115..11115 src has 3254..11115) v3 ==== 1715+0+0 (4177096548 0 0) 0x5b82400 con 0x10590000
  298. -558> 2014-06-02 11:16:18.255214 7f02026f0700 10 osd.123 11123 do_waiters -- start
  299. -557> 2014-06-02 11:16:18.255216 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  300. -556> 2014-06-02 11:16:18.255217 7f02026f0700 20 osd.123 11123 _dispatch 0x5b82400 osd_map(11115..11115 src has 3254..11115) v3
  301. -555> 2014-06-02 11:16:18.255220 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11115,11115], i have 11123, src has [3254,11115]
  302. -554> 2014-06-02 11:16:18.255222 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  303. -553> 2014-06-02 11:16:18.255224 7f02026f0700 10 osd.123 11123 do_waiters -- start
  304. -552> 2014-06-02 11:16:18.255225 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  305. -551> 2014-06-02 11:16:18.255228 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.33 192.168.1.103:6809/83017426 34 ==== pg_notify(14.103(732) epoch 11116) v5 ==== 40234+0+0 (4150600596 0 0) 0xa551c00 con 0x10eb5800
  306. -550> 2014-06-02 11:16:18.255235 7f02026f0700 10 osd.123 11123 do_waiters -- start
  307. -549> 2014-06-02 11:16:18.255237 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  308. -548> 2014-06-02 11:16:18.255238 7f02026f0700 20 osd.123 11123 _dispatch 0xa551c00 pg_notify(14.103(732) epoch 11116) v5
  309. -547> 2014-06-02 11:16:18.255241 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:06.988878, event: header_read, request: pg_notify(14.103(732) epoch 11116) v5
  310. -546> 2014-06-02 11:16:18.255246 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:06.988880, event: throttled, request: pg_notify(14.103(732) epoch 11116) v5
  311. -545> 2014-06-02 11:16:18.255249 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:06.990148, event: all_read, request: pg_notify(14.103(732) epoch 11116) v5
  312. -544> 2014-06-02 11:16:18.255252 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:18.255235, event: dispatched, request: pg_notify(14.103(732) epoch 11116) v5
  313. -543> 2014-06-02 11:16:18.255256 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:18.255256, event: waiting_for_osdmap, request: pg_notify(14.103(732) epoch 11116) v5
  314. -542> 2014-06-02 11:16:18.255259 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.33
  315. -541> 2014-06-02 11:16:18.255261 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xa551c00
  316. -540> 2014-06-02 11:16:18.255263 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(14.103(732) epoch 11116) v5
  317. -539> 2014-06-02 11:16:18.255266 7f02026f0700 5 -- op tracker -- , seq: 150790, time: 2014-06-02 11:16:18.255266, event: done, request: pg_notify(14.103(732) epoch 11116) v5
  318. -538> 2014-06-02 11:16:18.255272 7f02026f0700 10 osd.123 11123 do_waiters -- start
  319. -537> 2014-06-02 11:16:18.255274 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  320. -536> 2014-06-02 11:16:18.255365 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 27 ==== pg_notify(3.1b7(15) epoch 11113) v5 ==== 1515+0+0 (1470474718 0 0) 0x11d5a140 con 0x10c75c20
  321. -535> 2014-06-02 11:16:18.255377 7f02026f0700 10 osd.123 11123 do_waiters -- start
  322. -534> 2014-06-02 11:16:18.255379 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  323. -533> 2014-06-02 11:16:18.255380 7f02026f0700 20 osd.123 11123 _dispatch 0x11d5a140 pg_notify(3.1b7(15) epoch 11113) v5
  324. -532> 2014-06-02 11:16:18.255383 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:03.847638, event: header_read, request: pg_notify(3.1b7(15) epoch 11113) v5
  325. -531> 2014-06-02 11:16:18.255388 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:03.847641, event: throttled, request: pg_notify(3.1b7(15) epoch 11113) v5
  326. -530> 2014-06-02 11:16:18.255391 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:03.847749, event: all_read, request: pg_notify(3.1b7(15) epoch 11113) v5
  327. -529> 2014-06-02 11:16:18.255394 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:18.255376, event: dispatched, request: pg_notify(3.1b7(15) epoch 11113) v5
  328. -528> 2014-06-02 11:16:18.255400 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:18.255400, event: waiting_for_osdmap, request: pg_notify(3.1b7(15) epoch 11113) v5
  329. -527> 2014-06-02 11:16:18.255405 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  330. -526> 2014-06-02 11:16:18.255408 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11113 (i am 11123) 0x11d5a140
  331. -525> 2014-06-02 11:16:18.255410 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(15) epoch 11113) v5
  332. -524> 2014-06-02 11:16:18.255413 7f02026f0700 5 -- op tracker -- , seq: 150791, time: 2014-06-02 11:16:18.255413, event: done, request: pg_notify(3.1b7(15) epoch 11113) v5
  333. -523> 2014-06-02 11:16:18.255421 7f02026f0700 10 osd.123 11123 do_waiters -- start
  334. -522> 2014-06-02 11:16:18.255422 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  335. -521> 2014-06-02 11:16:18.255427 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.70 192.168.1.109:6811/18033379 21 ==== pg_notify(3.2d7(472) epoch 11116) v5 ==== 24242+0+0 (2854870479 0 0) 0x11a6e200 con 0x10c71080
  336. -520> 2014-06-02 11:16:18.255435 7f02026f0700 10 osd.123 11123 do_waiters -- start
  337. -519> 2014-06-02 11:16:18.255439 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  338. -518> 2014-06-02 11:16:18.255440 7f02026f0700 20 osd.123 11123 _dispatch 0x11a6e200 pg_notify(3.2d7(472) epoch 11116) v5
  339. -517> 2014-06-02 11:16:18.255443 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:06.969368, event: header_read, request: pg_notify(3.2d7(472) epoch 11116) v5
  340. -516> 2014-06-02 11:16:18.255447 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:06.969371, event: throttled, request: pg_notify(3.2d7(472) epoch 11116) v5
  341. -515> 2014-06-02 11:16:18.255450 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:06.970054, event: all_read, request: pg_notify(3.2d7(472) epoch 11116) v5
  342. -514> 2014-06-02 11:16:18.255453 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:18.255435, event: dispatched, request: pg_notify(3.2d7(472) epoch 11116) v5
  343. -513> 2014-06-02 11:16:18.255457 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:18.255457, event: waiting_for_osdmap, request: pg_notify(3.2d7(472) epoch 11116) v5
  344. -512> 2014-06-02 11:16:18.255461 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.70
  345. -511> 2014-06-02 11:16:18.255464 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x11a6e200
  346. -510> 2014-06-02 11:16:18.255466 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.2d7(472) epoch 11116) v5
  347. -509> 2014-06-02 11:16:18.255469 7f02026f0700 5 -- op tracker -- , seq: 150792, time: 2014-06-02 11:16:18.255469, event: done, request: pg_notify(3.2d7(472) epoch 11116) v5
  348. -508> 2014-06-02 11:16:18.255476 7f02026f0700 10 osd.123 11123 do_waiters -- start
  349. -507> 2014-06-02 11:16:18.255478 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  350. -506> 2014-06-02 11:16:18.255632 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.76 192.168.1.106:6807/95037986 106 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x1005ec00 con 0x2e87a60
  351. -505> 2014-06-02 11:16:18.255645 7f02026f0700 10 osd.123 11123 do_waiters -- start
  352. -504> 2014-06-02 11:16:18.255647 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  353. -503> 2014-06-02 11:16:18.255649 7f02026f0700 20 osd.123 11123 _dispatch 0x1005ec00 osd_map(11116..11116 src has 3254..11116) v3
  354. -502> 2014-06-02 11:16:18.255653 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  355. -501> 2014-06-02 11:16:18.255656 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  356. -500> 2014-06-02 11:16:18.255657 7f02026f0700 10 osd.123 11123 do_waiters -- start
  357. -499> 2014-06-02 11:16:18.255658 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  358. -498> 2014-06-02 11:16:18.255674 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 69 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x6637740 con 0x10e32100
  359. -497> 2014-06-02 11:16:18.255682 7f02026f0700 10 osd.123 11123 do_waiters -- start
  360. -496> 2014-06-02 11:16:18.255683 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  361. -495> 2014-06-02 11:16:18.255684 7f02026f0700 20 osd.123 11123 _dispatch 0x6637740 osd_map(11116..11116 src has 3254..11116) v3
  362. -494> 2014-06-02 11:16:18.255688 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  363. -493> 2014-06-02 11:16:18.255690 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  364. -492> 2014-06-02 11:16:18.255692 7f02026f0700 10 osd.123 11123 do_waiters -- start
  365. -491> 2014-06-02 11:16:18.255693 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  366. -490> 2014-06-02 11:16:18.255703 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 68 ==== pg_notify(15.64(9) epoch 11115) v5 ==== 1209+0+0 (1781814368 0 0) 0x90d9880 con 0x10589a20
  367. -489> 2014-06-02 11:16:18.255711 7f02026f0700 10 osd.123 11123 do_waiters -- start
  368. -488> 2014-06-02 11:16:18.255712 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  369. -487> 2014-06-02 11:16:18.255713 7f02026f0700 20 osd.123 11123 _dispatch 0x90d9880 pg_notify(15.64(9) epoch 11115) v5
  370. -486> 2014-06-02 11:16:18.255717 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:05.935886, event: header_read, request: pg_notify(15.64(9) epoch 11115) v5
  371. -485> 2014-06-02 11:16:18.255721 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:05.935887, event: throttled, request: pg_notify(15.64(9) epoch 11115) v5
  372. -484> 2014-06-02 11:16:18.255725 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:05.936055, event: all_read, request: pg_notify(15.64(9) epoch 11115) v5
  373. -483> 2014-06-02 11:16:18.255728 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:18.255710, event: dispatched, request: pg_notify(15.64(9) epoch 11115) v5
  374. -482> 2014-06-02 11:16:18.255732 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:18.255732, event: waiting_for_osdmap, request: pg_notify(15.64(9) epoch 11115) v5
  375. -481> 2014-06-02 11:16:18.255736 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  376. -480> 2014-06-02 11:16:18.255738 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x90d9880
  377. -479> 2014-06-02 11:16:18.255740 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.64(9) epoch 11115) v5
  378. -478> 2014-06-02 11:16:18.255743 7f02026f0700 5 -- op tracker -- , seq: 150793, time: 2014-06-02 11:16:18.255743, event: done, request: pg_notify(15.64(9) epoch 11115) v5
  379. -477> 2014-06-02 11:16:18.255750 7f02026f0700 10 osd.123 11123 do_waiters -- start
  380. -476> 2014-06-02 11:16:18.255752 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  381. -475> 2014-06-02 11:16:18.255756 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.22 192.168.1.101:6810/109034609 41 ==== pg_notify(15.d6(63) epoch 11116) v5 ==== 4091+0+0 (1424189528 0 0) 0x193a96c0 con 0x10590c60
  382. -474> 2014-06-02 11:16:18.255765 7f02026f0700 10 osd.123 11123 do_waiters -- start
  383. -473> 2014-06-02 11:16:18.255766 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  384. -472> 2014-06-02 11:16:18.255767 7f02026f0700 20 osd.123 11123 _dispatch 0x193a96c0 pg_notify(15.d6(63) epoch 11116) v5
  385. -471> 2014-06-02 11:16:18.255770 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:07.010404, event: header_read, request: pg_notify(15.d6(63) epoch 11116) v5
  386. -470> 2014-06-02 11:16:18.255774 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:07.010406, event: throttled, request: pg_notify(15.d6(63) epoch 11116) v5
  387. -469> 2014-06-02 11:16:18.255778 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:07.010576, event: all_read, request: pg_notify(15.d6(63) epoch 11116) v5
  388. -468> 2014-06-02 11:16:18.255781 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:18.255764, event: dispatched, request: pg_notify(15.d6(63) epoch 11116) v5
  389. -467> 2014-06-02 11:16:18.255785 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:18.255785, event: waiting_for_osdmap, request: pg_notify(15.d6(63) epoch 11116) v5
  390. -466> 2014-06-02 11:16:18.255789 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.22
  391. -465> 2014-06-02 11:16:18.255793 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x193a96c0
  392. -464> 2014-06-02 11:16:18.255795 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.d6(63) epoch 11116) v5
  393. -463> 2014-06-02 11:16:18.255798 7f02026f0700 5 -- op tracker -- , seq: 150794, time: 2014-06-02 11:16:18.255798, event: done, request: pg_notify(15.d6(63) epoch 11116) v5
  394. -462> 2014-06-02 11:16:18.255805 7f02026f0700 10 osd.123 11123 do_waiters -- start
  395. -461> 2014-06-02 11:16:18.255808 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  396. -460> 2014-06-02 11:16:18.255820 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 43 ==== pg_notify(16.6b1(530) epoch 11115) v5 ==== 28844+0+0 (2578870873 0 0) 0xe630380 con 0x10590000
  397. -459> 2014-06-02 11:16:18.255830 7f02026f0700 10 osd.123 11123 do_waiters -- start
  398. -458> 2014-06-02 11:16:18.255834 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  399. -457> 2014-06-02 11:16:18.255847 7f02026f0700 20 osd.123 11123 _dispatch 0xe630380 pg_notify(16.6b1(530) epoch 11115) v5
  400. -456> 2014-06-02 11:16:18.255850 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:06.025881, event: header_read, request: pg_notify(16.6b1(530) epoch 11115) v5
  401. -455> 2014-06-02 11:16:18.255854 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:06.025885, event: throttled, request: pg_notify(16.6b1(530) epoch 11115) v5
  402. -454> 2014-06-02 11:16:18.255857 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:06.026689, event: all_read, request: pg_notify(16.6b1(530) epoch 11115) v5
  403. -453> 2014-06-02 11:16:18.255860 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:18.255830, event: dispatched, request: pg_notify(16.6b1(530) epoch 11115) v5
  404. -452> 2014-06-02 11:16:18.255866 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:18.255866, event: waiting_for_osdmap, request: pg_notify(16.6b1(530) epoch 11115) v5
  405. -451> 2014-06-02 11:16:18.255871 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  406. -450> 2014-06-02 11:16:18.255873 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xe630380
  407. -449> 2014-06-02 11:16:18.255875 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.6b1(530) epoch 11115) v5
  408. -448> 2014-06-02 11:16:18.255878 7f02026f0700 5 -- op tracker -- , seq: 150795, time: 2014-06-02 11:16:18.255878, event: done, request: pg_notify(16.6b1(530) epoch 11115) v5
  409. -447> 2014-06-02 11:16:18.255884 7f02026f0700 10 osd.123 11123 do_waiters -- start
  410. -446> 2014-06-02 11:16:18.255886 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  411. -445> 2014-06-02 11:16:18.255964 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 28 ==== osd_map(11114..11114 src has 3254..11114) v3 ==== 3785+0+0 (1609340482 0 0) 0x12658240 con 0x10c75c20
  412. -444> 2014-06-02 11:16:18.255977 7f02026f0700 10 osd.123 11123 do_waiters -- start
  413. -443> 2014-06-02 11:16:18.255979 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  414. -442> 2014-06-02 11:16:18.255980 7f02026f0700 20 osd.123 11123 _dispatch 0x12658240 osd_map(11114..11114 src has 3254..11114) v3
  415. -441> 2014-06-02 11:16:18.255985 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11114,11114], i have 11123, src has [3254,11114]
  416. -440> 2014-06-02 11:16:18.255988 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  417. -439> 2014-06-02 11:16:18.255989 7f02026f0700 10 osd.123 11123 do_waiters -- start
  418. -438> 2014-06-02 11:16:18.255990 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  419. -437> 2014-06-02 11:16:18.255994 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.70 192.168.1.109:6811/18033379 22 ==== pg_notify(2.176(557) epoch 11116) v5 ==== 29305+0+0 (105024511 0 0) 0x11a6a840 con 0x10c71080
  420. -436> 2014-06-02 11:16:18.256003 7f02026f0700 10 osd.123 11123 do_waiters -- start
  421. -435> 2014-06-02 11:16:18.256004 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  422. -434> 2014-06-02 11:16:18.256005 7f02026f0700 20 osd.123 11123 _dispatch 0x11a6a840 pg_notify(2.176(557) epoch 11116) v5
  423. -433> 2014-06-02 11:16:18.256009 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:06.997455, event: header_read, request: pg_notify(2.176(557) epoch 11116) v5
  424. -432> 2014-06-02 11:16:18.256013 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:06.997458, event: throttled, request: pg_notify(2.176(557) epoch 11116) v5
  425. -431> 2014-06-02 11:16:18.256017 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:06.998256, event: all_read, request: pg_notify(2.176(557) epoch 11116) v5
  426. -430> 2014-06-02 11:16:18.256020 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:18.256002, event: dispatched, request: pg_notify(2.176(557) epoch 11116) v5
  427. -429> 2014-06-02 11:16:18.256024 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:18.256024, event: waiting_for_osdmap, request: pg_notify(2.176(557) epoch 11116) v5
  428. -428> 2014-06-02 11:16:18.256028 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.70
  429. -427> 2014-06-02 11:16:18.256030 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x11a6a840
  430. -426> 2014-06-02 11:16:18.256033 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(2.176(557) epoch 11116) v5
  431. -425> 2014-06-02 11:16:18.256037 7f02026f0700 5 -- op tracker -- , seq: 150796, time: 2014-06-02 11:16:18.256037, event: done, request: pg_notify(2.176(557) epoch 11116) v5
  432. -424> 2014-06-02 11:16:18.256045 7f02026f0700 10 osd.123 11123 do_waiters -- start
  433. -423> 2014-06-02 11:16:18.256058 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  434. -422> 2014-06-02 11:16:18.256129 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.76 192.168.1.106:6807/95037986 107 ==== pg_notify(15.279(19) epoch 11116) v5 ==== 1735+0+0 (2715212458 0 0) 0xe632d80 con 0x2e87a60
  435. -421> 2014-06-02 11:16:18.256141 7f02026f0700 10 osd.123 11123 do_waiters -- start
  436. -420> 2014-06-02 11:16:18.256142 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  437. -419> 2014-06-02 11:16:18.256144 7f02026f0700 20 osd.123 11123 _dispatch 0xe632d80 pg_notify(15.279(19) epoch 11116) v5
  438. -418> 2014-06-02 11:16:18.256147 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:07.063813, event: header_read, request: pg_notify(15.279(19) epoch 11116) v5
  439. -417> 2014-06-02 11:16:18.256156 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:07.063813, event: throttled, request: pg_notify(15.279(19) epoch 11116) v5
  440. -416> 2014-06-02 11:16:18.256159 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:07.063893, event: all_read, request: pg_notify(15.279(19) epoch 11116) v5
  441. -415> 2014-06-02 11:16:18.256165 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:18.256140, event: dispatched, request: pg_notify(15.279(19) epoch 11116) v5
  442. -414> 2014-06-02 11:16:18.256171 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:18.256171, event: waiting_for_osdmap, request: pg_notify(15.279(19) epoch 11116) v5
  443. -413> 2014-06-02 11:16:18.256176 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.76
  444. -412> 2014-06-02 11:16:18.256178 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xe632d80
  445. -411> 2014-06-02 11:16:18.256180 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.279(19) epoch 11116) v5
  446. -410> 2014-06-02 11:16:18.256183 7f02026f0700 5 -- op tracker -- , seq: 150797, time: 2014-06-02 11:16:18.256183, event: done, request: pg_notify(15.279(19) epoch 11116) v5
  447. -409> 2014-06-02 11:16:18.256191 7f02026f0700 10 osd.123 11123 do_waiters -- start
  448. -408> 2014-06-02 11:16:18.256193 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  449. -407> 2014-06-02 11:16:18.256199 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 70 ==== pg_notify(16.4b(14) epoch 11116) v5 ==== 1440+0+0 (3126231414 0 0) 0xa552300 con 0x10e32100
  450. -406> 2014-06-02 11:16:18.256209 7f02026f0700 10 osd.123 11123 do_waiters -- start
  451. -405> 2014-06-02 11:16:18.256210 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  452. -404> 2014-06-02 11:16:18.256212 7f02026f0700 20 osd.123 11123 _dispatch 0xa552300 pg_notify(16.4b(14) epoch 11116) v5
  453. -403> 2014-06-02 11:16:18.256215 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:06.922275, event: header_read, request: pg_notify(16.4b(14) epoch 11116) v5
  454. -402> 2014-06-02 11:16:18.256219 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:06.922276, event: throttled, request: pg_notify(16.4b(14) epoch 11116) v5
  455. -401> 2014-06-02 11:16:18.256222 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:06.922420, event: all_read, request: pg_notify(16.4b(14) epoch 11116) v5
  456. -400> 2014-06-02 11:16:18.256225 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:18.256208, event: dispatched, request: pg_notify(16.4b(14) epoch 11116) v5
  457. -399> 2014-06-02 11:16:18.256229 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:18.256229, event: waiting_for_osdmap, request: pg_notify(16.4b(14) epoch 11116) v5
  458. -398> 2014-06-02 11:16:18.256234 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.96
  459. -397> 2014-06-02 11:16:18.256239 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xa552300
  460. -396> 2014-06-02 11:16:18.256243 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.4b(14) epoch 11116) v5
  461. -395> 2014-06-02 11:16:18.256246 7f02026f0700 5 -- op tracker -- , seq: 150798, time: 2014-06-02 11:16:18.256246, event: done, request: pg_notify(16.4b(14) epoch 11116) v5
  462. -394> 2014-06-02 11:16:18.256252 7f02026f0700 10 osd.123 11123 do_waiters -- start
  463. -393> 2014-06-02 11:16:18.256253 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  464. -392> 2014-06-02 11:16:18.256258 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 69 ==== pg_notify(8.127(13) epoch 11115) v5 ==== 1397+0+0 (779633999 0 0) 0x90db640 con 0x10589a20
  465. -391> 2014-06-02 11:16:18.256266 7f02026f0700 10 osd.123 11123 do_waiters -- start
  466. -390> 2014-06-02 11:16:18.256267 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  467. -389> 2014-06-02 11:16:18.256268 7f02026f0700 20 osd.123 11123 _dispatch 0x90db640 pg_notify(8.127(13) epoch 11115) v5
  468. -388> 2014-06-02 11:16:18.256271 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:06.029871, event: header_read, request: pg_notify(8.127(13) epoch 11115) v5
  469. -387> 2014-06-02 11:16:18.256277 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:06.029873, event: throttled, request: pg_notify(8.127(13) epoch 11115) v5
  470. -386> 2014-06-02 11:16:18.256280 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:06.029985, event: all_read, request: pg_notify(8.127(13) epoch 11115) v5
  471. -385> 2014-06-02 11:16:18.256295 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:18.256266, event: dispatched, request: pg_notify(8.127(13) epoch 11115) v5
  472. -384> 2014-06-02 11:16:18.256299 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:18.256299, event: waiting_for_osdmap, request: pg_notify(8.127(13) epoch 11115) v5
  473. -383> 2014-06-02 11:16:18.256303 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  474. -382> 2014-06-02 11:16:18.256305 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x90db640
  475. -381> 2014-06-02 11:16:18.256308 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(8.127(13) epoch 11115) v5
  476. -380> 2014-06-02 11:16:18.256310 7f02026f0700 5 -- op tracker -- , seq: 150799, time: 2014-06-02 11:16:18.256310, event: done, request: pg_notify(8.127(13) epoch 11115) v5
  477. -379> 2014-06-02 11:16:18.256316 7f02026f0700 10 osd.123 11123 do_waiters -- start
  478. -378> 2014-06-02 11:16:18.256318 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  479. -377> 2014-06-02 11:16:18.256324 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.22 192.168.1.101:6810/109034609 42 ==== pg_notify(16.dc8(713) epoch 11116) v5 ==== 38273+0+0 (2301607852 0 0) 0x193ac7c0 con 0x10590c60
  480. -376> 2014-06-02 11:16:18.256332 7f02026f0700 10 osd.123 11123 do_waiters -- start
  481. -375> 2014-06-02 11:16:18.256334 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  482. -374> 2014-06-02 11:16:18.256335 7f02026f0700 20 osd.123 11123 _dispatch 0x193ac7c0 pg_notify(16.dc8(713) epoch 11116) v5
  483. -373> 2014-06-02 11:16:18.256338 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:07.020804, event: header_read, request: pg_notify(16.dc8(713) epoch 11116) v5
  484. -372> 2014-06-02 11:16:18.256342 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:07.020806, event: throttled, request: pg_notify(16.dc8(713) epoch 11116) v5
  485. -371> 2014-06-02 11:16:18.256345 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:07.021868, event: all_read, request: pg_notify(16.dc8(713) epoch 11116) v5
  486. -370> 2014-06-02 11:16:18.256348 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:18.256332, event: dispatched, request: pg_notify(16.dc8(713) epoch 11116) v5
  487. -369> 2014-06-02 11:16:18.256352 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:18.256352, event: waiting_for_osdmap, request: pg_notify(16.dc8(713) epoch 11116) v5
  488. -368> 2014-06-02 11:16:18.256356 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.22
  489. -367> 2014-06-02 11:16:18.256358 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x193ac7c0
  490. -366> 2014-06-02 11:16:18.256360 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.dc8(713) epoch 11116) v5
  491. -365> 2014-06-02 11:16:18.256363 7f02026f0700 5 -- op tracker -- , seq: 150800, time: 2014-06-02 11:16:18.256363, event: done, request: pg_notify(16.dc8(713) epoch 11116) v5
  492. -364> 2014-06-02 11:16:18.256369 7f02026f0700 10 osd.123 11123 do_waiters -- start
  493. -363> 2014-06-02 11:16:18.256371 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  494. -362> 2014-06-02 11:16:18.256473 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 44 ==== pg_notify(9.2fe(550) epoch 11115) v5 ==== 28872+0+0 (2711430769 0 0) 0xe637540 con 0x10590000
  495. -361> 2014-06-02 11:16:18.256485 7f02026f0700 10 osd.123 11123 do_waiters -- start
  496. -360> 2014-06-02 11:16:18.256487 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  497. -359> 2014-06-02 11:16:18.256488 7f02026f0700 20 osd.123 11123 _dispatch 0xe637540 pg_notify(9.2fe(550) epoch 11115) v5
  498. -358> 2014-06-02 11:16:18.256491 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:06.068967, event: header_read, request: pg_notify(9.2fe(550) epoch 11115) v5
  499. -357> 2014-06-02 11:16:18.256496 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:06.068969, event: throttled, request: pg_notify(9.2fe(550) epoch 11115) v5
  500. -356> 2014-06-02 11:16:18.256499 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:06.069861, event: all_read, request: pg_notify(9.2fe(550) epoch 11115) v5
  501. -355> 2014-06-02 11:16:18.256503 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:18.256485, event: dispatched, request: pg_notify(9.2fe(550) epoch 11115) v5
  502. -354> 2014-06-02 11:16:18.256509 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:18.256509, event: waiting_for_osdmap, request: pg_notify(9.2fe(550) epoch 11115) v5
  503. -353> 2014-06-02 11:16:18.256514 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  504. -352> 2014-06-02 11:16:18.256516 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xe637540
  505. -351> 2014-06-02 11:16:18.256518 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(9.2fe(550) epoch 11115) v5
  506. -350> 2014-06-02 11:16:18.256521 7f02026f0700 5 -- op tracker -- , seq: 150801, time: 2014-06-02 11:16:18.256521, event: done, request: pg_notify(9.2fe(550) epoch 11115) v5
  507. -349> 2014-06-02 11:16:18.256528 7f02026f0700 10 osd.123 11123 do_waiters -- start
  508. -348> 2014-06-02 11:16:18.256529 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  509. -347> 2014-06-02 11:16:18.256598 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 29 ==== pg_notify(13.330(17) epoch 11114) v5 ==== 1633+0+0 (2484041878 0 0) 0x90df1c0 con 0x10c75c20
  510. -346> 2014-06-02 11:16:18.256610 7f02026f0700 10 osd.123 11123 do_waiters -- start
  511. -345> 2014-06-02 11:16:18.256612 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  512. -344> 2014-06-02 11:16:18.256613 7f02026f0700 20 osd.123 11123 _dispatch 0x90df1c0 pg_notify(13.330(17) epoch 11114) v5
  513. -343> 2014-06-02 11:16:18.256616 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:04.656228, event: header_read, request: pg_notify(13.330(17) epoch 11114) v5
  514. -342> 2014-06-02 11:16:18.256621 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:04.656230, event: throttled, request: pg_notify(13.330(17) epoch 11114) v5
  515. -341> 2014-06-02 11:16:18.256624 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:04.656387, event: all_read, request: pg_notify(13.330(17) epoch 11114) v5
  516. -340> 2014-06-02 11:16:18.256627 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:18.256610, event: dispatched, request: pg_notify(13.330(17) epoch 11114) v5
  517. -339> 2014-06-02 11:16:18.256633 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:18.256633, event: waiting_for_osdmap, request: pg_notify(13.330(17) epoch 11114) v5
  518. -338> 2014-06-02 11:16:18.256638 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  519. -337> 2014-06-02 11:16:18.256641 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0x90df1c0
  520. -336> 2014-06-02 11:16:18.256643 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(13.330(17) epoch 11114) v5
  521. -335> 2014-06-02 11:16:18.256646 7f02026f0700 5 -- op tracker -- , seq: 150802, time: 2014-06-02 11:16:18.256646, event: done, request: pg_notify(13.330(17) epoch 11114) v5
  522. -334> 2014-06-02 11:16:18.256652 7f02026f0700 10 osd.123 11123 do_waiters -- start
  523. -333> 2014-06-02 11:16:18.256654 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  524. -332> 2014-06-02 11:16:18.256660 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.70 192.168.1.109:6811/18033379 23 ==== pg_notify(10.19f(489),13.1be(32) epoch 11116) v5 ==== 28023+0+0 (2994849824 0 0) 0x11a6e580 con 0x10c71080
  525. -331> 2014-06-02 11:16:18.256669 7f02026f0700 10 osd.123 11123 do_waiters -- start
  526. -330> 2014-06-02 11:16:18.256672 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  527. -329> 2014-06-02 11:16:18.256674 7f02026f0700 20 osd.123 11123 _dispatch 0x11a6e580 pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  528. -328> 2014-06-02 11:16:18.256677 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:07.002955, event: header_read, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  529. -327> 2014-06-02 11:16:18.256681 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:07.002958, event: throttled, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  530. -326> 2014-06-02 11:16:18.256685 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:07.003749, event: all_read, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  531. -325> 2014-06-02 11:16:18.256689 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:18.256669, event: dispatched, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  532. -324> 2014-06-02 11:16:18.256694 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:18.256694, event: waiting_for_osdmap, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  533. -323> 2014-06-02 11:16:18.256700 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.70
  534. -322> 2014-06-02 11:16:18.256704 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x11a6e580
  535. -321> 2014-06-02 11:16:18.256706 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  536. -320> 2014-06-02 11:16:18.256709 7f02026f0700 5 -- op tracker -- , seq: 150803, time: 2014-06-02 11:16:18.256709, event: done, request: pg_notify(10.19f(489),13.1be(32) epoch 11116) v5
  537. -319> 2014-06-02 11:16:18.256717 7f02026f0700 10 osd.123 11123 do_waiters -- start
  538. -318> 2014-06-02 11:16:18.256718 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  539. -317> 2014-06-02 11:16:18.256890 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.76 192.168.1.106:6807/95037986 108 ==== pg_notify(3.1b7(563) epoch 11116) v5 ==== 30423+0+0 (2978780871 0 0) 0x76a5940 con 0x2e87a60
  540. -316> 2014-06-02 11:16:18.256903 7f02026f0700 10 osd.123 11123 do_waiters -- start
  541. -315> 2014-06-02 11:16:18.256905 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  542. -314> 2014-06-02 11:16:18.256907 7f02026f0700 20 osd.123 11123 _dispatch 0x76a5940 pg_notify(3.1b7(563) epoch 11116) v5
  543. -313> 2014-06-02 11:16:18.256910 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:07.105366, event: header_read, request: pg_notify(3.1b7(563) epoch 11116) v5
  544. -312> 2014-06-02 11:16:18.256915 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:07.105370, event: throttled, request: pg_notify(3.1b7(563) epoch 11116) v5
  545. -311> 2014-06-02 11:16:18.256918 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:07.106258, event: all_read, request: pg_notify(3.1b7(563) epoch 11116) v5
  546. -310> 2014-06-02 11:16:18.256921 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:18.256903, event: dispatched, request: pg_notify(3.1b7(563) epoch 11116) v5
  547. -309> 2014-06-02 11:16:18.256927 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:18.256927, event: waiting_for_osdmap, request: pg_notify(3.1b7(563) epoch 11116) v5
  548. -308> 2014-06-02 11:16:18.256932 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.76
  549. -307> 2014-06-02 11:16:18.256935 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x76a5940
  550. -306> 2014-06-02 11:16:18.256937 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(563) epoch 11116) v5
  551. -305> 2014-06-02 11:16:18.256940 7f02026f0700 5 -- op tracker -- , seq: 150804, time: 2014-06-02 11:16:18.256940, event: done, request: pg_notify(3.1b7(563) epoch 11116) v5
  552. -304> 2014-06-02 11:16:18.256947 7f02026f0700 10 osd.123 11123 do_waiters -- start
  553. -303> 2014-06-02 11:16:18.256948 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  554. -302> 2014-06-02 11:16:18.257019 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.96 192.168.1.107:6805/16019917 71 ==== pg_notify(5.393(523) epoch 11116) v5 ==== 26775+0+0 (1074736097 0 0) 0x11a68c40 con 0x10e32100
  555. -301> 2014-06-02 11:16:18.257031 7f02026f0700 10 osd.123 11123 do_waiters -- start
  556. -300> 2014-06-02 11:16:18.257034 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  557. -299> 2014-06-02 11:16:18.257036 7f02026f0700 20 osd.123 11123 _dispatch 0x11a68c40 pg_notify(5.393(523) epoch 11116) v5
  558. -298> 2014-06-02 11:16:18.257039 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:06.955274, event: header_read, request: pg_notify(5.393(523) epoch 11116) v5
  559. -297> 2014-06-02 11:16:18.257043 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:06.955277, event: throttled, request: pg_notify(5.393(523) epoch 11116) v5
  560. -296> 2014-06-02 11:16:18.257047 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:06.956027, event: all_read, request: pg_notify(5.393(523) epoch 11116) v5
  561. -295> 2014-06-02 11:16:18.257050 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:18.257031, event: dispatched, request: pg_notify(5.393(523) epoch 11116) v5
  562. -294> 2014-06-02 11:16:18.257056 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:18.257055, event: waiting_for_osdmap, request: pg_notify(5.393(523) epoch 11116) v5
  563. -293> 2014-06-02 11:16:18.257061 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.96
  564. -292> 2014-06-02 11:16:18.257062 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x11a68c40
  565. -291> 2014-06-02 11:16:18.257065 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(5.393(523) epoch 11116) v5
  566. -290> 2014-06-02 11:16:18.257067 7f02026f0700 5 -- op tracker -- , seq: 150805, time: 2014-06-02 11:16:18.257067, event: done, request: pg_notify(5.393(523) epoch 11116) v5
  567. -289> 2014-06-02 11:16:18.257075 7f02026f0700 10 osd.123 11123 do_waiters -- start
  568. -288> 2014-06-02 11:16:18.257088 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  569. -287> 2014-06-02 11:16:18.257175 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 70 ==== pg_notify(6.1bf(9) epoch 11115) v5 ==== 1209+0+0 (3076910962 0 0) 0x90daf40 con 0x10589a20
  570. -286> 2014-06-02 11:16:18.257187 7f02026f0700 10 osd.123 11123 do_waiters -- start
  571. -285> 2014-06-02 11:16:18.257189 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  572. -284> 2014-06-02 11:16:18.257190 7f02026f0700 20 osd.123 11123 _dispatch 0x90daf40 pg_notify(6.1bf(9) epoch 11115) v5
  573. -283> 2014-06-02 11:16:18.257193 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:06.055453, event: header_read, request: pg_notify(6.1bf(9) epoch 11115) v5
  574. -282> 2014-06-02 11:16:18.257198 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:06.055455, event: throttled, request: pg_notify(6.1bf(9) epoch 11115) v5
  575. -281> 2014-06-02 11:16:18.257201 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:06.055544, event: all_read, request: pg_notify(6.1bf(9) epoch 11115) v5
  576. -280> 2014-06-02 11:16:18.257204 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:18.257186, event: dispatched, request: pg_notify(6.1bf(9) epoch 11115) v5
  577. -279> 2014-06-02 11:16:18.257208 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:18.257208, event: waiting_for_osdmap, request: pg_notify(6.1bf(9) epoch 11115) v5
  578. -278> 2014-06-02 11:16:18.257213 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  579. -277> 2014-06-02 11:16:18.257216 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x90daf40
  580. -276> 2014-06-02 11:16:18.257218 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1bf(9) epoch 11115) v5
  581. -275> 2014-06-02 11:16:18.257221 7f02026f0700 5 -- op tracker -- , seq: 150806, time: 2014-06-02 11:16:18.257221, event: done, request: pg_notify(6.1bf(9) epoch 11115) v5
  582. -274> 2014-06-02 11:16:18.257240 7f02026f0700 10 osd.123 11123 do_waiters -- start
  583. -273> 2014-06-02 11:16:18.257242 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  584. -272> 2014-06-02 11:16:18.257246 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 45 ==== pg_notify(6.1b8(14) epoch 11115) v5 ==== 1480+0+0 (1967055265 0 0) 0xe633640 con 0x10590000
  585. -271> 2014-06-02 11:16:18.257254 7f02026f0700 10 osd.123 11123 do_waiters -- start
  586. -270> 2014-06-02 11:16:18.257260 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  587. -269> 2014-06-02 11:16:18.257261 7f02026f0700 20 osd.123 11123 _dispatch 0xe633640 pg_notify(6.1b8(14) epoch 11115) v5
  588. -268> 2014-06-02 11:16:18.257264 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:06.088979, event: header_read, request: pg_notify(6.1b8(14) epoch 11115) v5
  589. -267> 2014-06-02 11:16:18.257269 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:06.088982, event: throttled, request: pg_notify(6.1b8(14) epoch 11115) v5
  590. -266> 2014-06-02 11:16:18.257272 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:06.089089, event: all_read, request: pg_notify(6.1b8(14) epoch 11115) v5
  591. -265> 2014-06-02 11:16:18.257276 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:18.257254, event: dispatched, request: pg_notify(6.1b8(14) epoch 11115) v5
  592. -264> 2014-06-02 11:16:18.257281 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:18.257281, event: waiting_for_osdmap, request: pg_notify(6.1b8(14) epoch 11115) v5
  593. -263> 2014-06-02 11:16:18.257287 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  594. -262> 2014-06-02 11:16:18.257290 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0xe633640
  595. -261> 2014-06-02 11:16:18.257292 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1b8(14) epoch 11115) v5
  596. -260> 2014-06-02 11:16:18.257295 7f02026f0700 5 -- op tracker -- , seq: 150807, time: 2014-06-02 11:16:18.257295, event: done, request: pg_notify(6.1b8(14) epoch 11115) v5
  597. -259> 2014-06-02 11:16:18.257301 7f02026f0700 10 osd.123 11123 do_waiters -- start
  598. -258> 2014-06-02 11:16:18.257303 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  599. -257> 2014-06-02 11:16:18.257308 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 30 ==== pg_notify(15.207(617) epoch 11114) v5 ==== 32585+0+0 (907444251 0 0) 0x90db2c0 con 0x10c75c20
  600. -256> 2014-06-02 11:16:18.257316 7f02026f0700 10 osd.123 11123 do_waiters -- start
  601. -255> 2014-06-02 11:16:18.257317 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  602. -254> 2014-06-02 11:16:18.257319 7f02026f0700 20 osd.123 11123 _dispatch 0x90db2c0 pg_notify(15.207(617) epoch 11114) v5
  603. -253> 2014-06-02 11:16:18.257321 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:04.735762, event: header_read, request: pg_notify(15.207(617) epoch 11114) v5
  604. -252> 2014-06-02 11:16:18.257326 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:04.735765, event: throttled, request: pg_notify(15.207(617) epoch 11114) v5
  605. -251> 2014-06-02 11:16:18.257331 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:04.736749, event: all_read, request: pg_notify(15.207(617) epoch 11114) v5
  606. -250> 2014-06-02 11:16:18.257334 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:18.257316, event: dispatched, request: pg_notify(15.207(617) epoch 11114) v5
  607. -249> 2014-06-02 11:16:18.257340 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:18.257340, event: waiting_for_osdmap, request: pg_notify(15.207(617) epoch 11114) v5
  608. -248> 2014-06-02 11:16:18.257346 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  609. -247> 2014-06-02 11:16:18.257347 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0x90db2c0
  610. -246> 2014-06-02 11:16:18.257350 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.207(617) epoch 11114) v5
  611. -245> 2014-06-02 11:16:18.257353 7f02026f0700 5 -- op tracker -- , seq: 150808, time: 2014-06-02 11:16:18.257352, event: done, request: pg_notify(15.207(617) epoch 11114) v5
  612. -244> 2014-06-02 11:16:18.257359 7f02026f0700 10 osd.123 11123 do_waiters -- start
  613. -243> 2014-06-02 11:16:18.257360 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  614. -242> 2014-06-02 11:16:18.257433 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 71 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x5bb5100 con 0x10589a20
  615. -241> 2014-06-02 11:16:18.257445 7f02026f0700 10 osd.123 11123 do_waiters -- start
  616. -240> 2014-06-02 11:16:18.257447 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  617. -239> 2014-06-02 11:16:18.257449 7f02026f0700 20 osd.123 11123 _dispatch 0x5bb5100 osd_map(11116..11116 src has 3254..11116) v3
  618. -238> 2014-06-02 11:16:18.257453 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  619. -237> 2014-06-02 11:16:18.257456 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  620. -236> 2014-06-02 11:16:18.257458 7f02026f0700 10 osd.123 11123 do_waiters -- start
  621. -235> 2014-06-02 11:16:18.257459 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  622. -234> 2014-06-02 11:16:18.257463 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 46 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x5b821c0 con 0x10590000
  623. -233> 2014-06-02 11:16:18.257471 7f02026f0700 10 osd.123 11123 do_waiters -- start
  624. -232> 2014-06-02 11:16:18.257472 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  625. -231> 2014-06-02 11:16:18.257473 7f02026f0700 20 osd.123 11123 _dispatch 0x5b821c0 osd_map(11116..11116 src has 3254..11116) v3
  626. -230> 2014-06-02 11:16:18.257477 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  627. -229> 2014-06-02 11:16:18.257479 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  628. -228> 2014-06-02 11:16:18.257481 7f02026f0700 10 osd.123 11123 do_waiters -- start
  629. -227> 2014-06-02 11:16:18.257482 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  630. -226> 2014-06-02 11:16:18.257485 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 31 ==== pg_notify(3.1b7(15) epoch 11114) v5 ==== 1515+0+0 (3708971475 0 0) 0x90d9a40 con 0x10c75c20
  631. -225> 2014-06-02 11:16:18.257493 7f02026f0700 10 osd.123 11123 do_waiters -- start
  632. -224> 2014-06-02 11:16:18.257494 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  633. -223> 2014-06-02 11:16:18.257495 7f02026f0700 20 osd.123 11123 _dispatch 0x90d9a40 pg_notify(3.1b7(15) epoch 11114) v5
  634. -222> 2014-06-02 11:16:18.257498 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:04.827584, event: header_read, request: pg_notify(3.1b7(15) epoch 11114) v5
  635. -221> 2014-06-02 11:16:18.257503 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:04.827586, event: throttled, request: pg_notify(3.1b7(15) epoch 11114) v5
  636. -220> 2014-06-02 11:16:18.257506 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:04.827692, event: all_read, request: pg_notify(3.1b7(15) epoch 11114) v5
  637. -219> 2014-06-02 11:16:18.257510 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:18.257492, event: dispatched, request: pg_notify(3.1b7(15) epoch 11114) v5
  638. -218> 2014-06-02 11:16:18.257513 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:18.257513, event: waiting_for_osdmap, request: pg_notify(3.1b7(15) epoch 11114) v5
  639. -217> 2014-06-02 11:16:18.257517 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  640. -216> 2014-06-02 11:16:18.257520 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11114 (i am 11123) 0x90d9a40
  641. -215> 2014-06-02 11:16:18.257522 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(15) epoch 11114) v5
  642. -214> 2014-06-02 11:16:18.257525 7f02026f0700 5 -- op tracker -- , seq: 150809, time: 2014-06-02 11:16:18.257525, event: done, request: pg_notify(3.1b7(15) epoch 11114) v5
  643. -213> 2014-06-02 11:16:18.257531 7f02026f0700 10 osd.123 11123 do_waiters -- start
  644. -212> 2014-06-02 11:16:18.257533 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  645. -211> 2014-06-02 11:16:18.257538 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 72 ==== pg_notify(15.64(9) epoch 11116) v5 ==== 1209+0+0 (2823862582 0 0) 0x90dc0c0 con 0x10589a20
  646. -210> 2014-06-02 11:16:18.257545 7f02026f0700 10 osd.123 11123 do_waiters -- start
  647. -209> 2014-06-02 11:16:18.257547 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  648. -208> 2014-06-02 11:16:18.257548 7f02026f0700 20 osd.123 11123 _dispatch 0x90dc0c0 pg_notify(15.64(9) epoch 11116) v5
  649. -207> 2014-06-02 11:16:18.257551 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:06.911145, event: header_read, request: pg_notify(15.64(9) epoch 11116) v5
  650. -206> 2014-06-02 11:16:18.257555 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:06.911146, event: throttled, request: pg_notify(15.64(9) epoch 11116) v5
  651. -205> 2014-06-02 11:16:18.257558 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:06.911302, event: all_read, request: pg_notify(15.64(9) epoch 11116) v5
  652. -204> 2014-06-02 11:16:18.257561 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:18.257545, event: dispatched, request: pg_notify(15.64(9) epoch 11116) v5
  653. -203> 2014-06-02 11:16:18.257565 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:18.257565, event: waiting_for_osdmap, request: pg_notify(15.64(9) epoch 11116) v5
  654. -202> 2014-06-02 11:16:18.257569 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  655. -201> 2014-06-02 11:16:18.257571 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x90dc0c0
  656. -200> 2014-06-02 11:16:18.257573 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.64(9) epoch 11116) v5
  657. -199> 2014-06-02 11:16:18.257576 7f02026f0700 5 -- op tracker -- , seq: 150810, time: 2014-06-02 11:16:18.257575, event: done, request: pg_notify(15.64(9) epoch 11116) v5
  658. -198> 2014-06-02 11:16:18.257582 7f02026f0700 10 osd.123 11123 do_waiters -- start
  659. -197> 2014-06-02 11:16:18.257595 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  660. -196> 2014-06-02 11:16:18.257599 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 47 ==== pg_notify(16.6b1(530) epoch 11116) v5 ==== 28844+0+0 (726488187 0 0) 0xe631f80 con 0x10590000
  661. -195> 2014-06-02 11:16:18.257606 7f02026f0700 10 osd.123 11123 do_waiters -- start
  662. -194> 2014-06-02 11:16:18.257608 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  663. -193> 2014-06-02 11:16:18.257609 7f02026f0700 20 osd.123 11123 _dispatch 0xe631f80 pg_notify(16.6b1(530) epoch 11116) v5
  664. -192> 2014-06-02 11:16:18.257612 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:06.973320, event: header_read, request: pg_notify(16.6b1(530) epoch 11116) v5
  665. -191> 2014-06-02 11:16:18.257615 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:06.973322, event: throttled, request: pg_notify(16.6b1(530) epoch 11116) v5
  666. -190> 2014-06-02 11:16:18.257619 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:06.974100, event: all_read, request: pg_notify(16.6b1(530) epoch 11116) v5
  667. -189> 2014-06-02 11:16:18.257622 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:18.257606, event: dispatched, request: pg_notify(16.6b1(530) epoch 11116) v5
  668. -188> 2014-06-02 11:16:18.257625 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:18.257625, event: waiting_for_osdmap, request: pg_notify(16.6b1(530) epoch 11116) v5
  669. -187> 2014-06-02 11:16:18.257631 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  670. -186> 2014-06-02 11:16:18.257633 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xe631f80
  671. -185> 2014-06-02 11:16:18.257635 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(16.6b1(530) epoch 11116) v5
  672. -184> 2014-06-02 11:16:18.257638 7f02026f0700 5 -- op tracker -- , seq: 150811, time: 2014-06-02 11:16:18.257638, event: done, request: pg_notify(16.6b1(530) epoch 11116) v5
  673. -183> 2014-06-02 11:16:18.257657 7f02026f0700 10 osd.123 11123 do_waiters -- start
  674. -182> 2014-06-02 11:16:18.257659 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  675. -181> 2014-06-02 11:16:18.257853 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 32 ==== osd_map(11115..11115 src has 3254..11115) v3 ==== 1715+0+0 (4177096548 0 0) 0x5a2f2c0 con 0x10c75c20
  676. -180> 2014-06-02 11:16:18.257866 7f02026f0700 10 osd.123 11123 do_waiters -- start
  677. -179> 2014-06-02 11:16:18.257868 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  678. -178> 2014-06-02 11:16:18.257869 7f02026f0700 20 osd.123 11123 _dispatch 0x5a2f2c0 osd_map(11115..11115 src has 3254..11115) v3
  679. -177> 2014-06-02 11:16:18.257874 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11115,11115], i have 11123, src has [3254,11115]
  680. -176> 2014-06-02 11:16:18.257876 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  681. -175> 2014-06-02 11:16:18.257878 7f02026f0700 10 osd.123 11123 do_waiters -- start
  682. -174> 2014-06-02 11:16:18.257879 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  683. -173> 2014-06-02 11:16:18.257883 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 73 ==== pg_notify(8.127(13) epoch 11116) v5 ==== 1397+0+0 (3916713185 0 0) 0x90db800 con 0x10589a20
  684. -172> 2014-06-02 11:16:18.257891 7f02026f0700 10 osd.123 11123 do_waiters -- start
  685. -171> 2014-06-02 11:16:18.257892 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  686. -170> 2014-06-02 11:16:18.257893 7f02026f0700 20 osd.123 11123 _dispatch 0x90db800 pg_notify(8.127(13) epoch 11116) v5
  687. -169> 2014-06-02 11:16:18.257896 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:06.993859, event: header_read, request: pg_notify(8.127(13) epoch 11116) v5
  688. -168> 2014-06-02 11:16:18.257903 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:06.993861, event: throttled, request: pg_notify(8.127(13) epoch 11116) v5
  689. -167> 2014-06-02 11:16:18.257906 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:06.993975, event: all_read, request: pg_notify(8.127(13) epoch 11116) v5
  690. -166> 2014-06-02 11:16:18.257910 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:18.257891, event: dispatched, request: pg_notify(8.127(13) epoch 11116) v5
  691. -165> 2014-06-02 11:16:18.257913 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:18.257913, event: waiting_for_osdmap, request: pg_notify(8.127(13) epoch 11116) v5
  692. -164> 2014-06-02 11:16:18.257917 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  693. -163> 2014-06-02 11:16:18.257919 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x90db800
  694. -162> 2014-06-02 11:16:18.257921 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(8.127(13) epoch 11116) v5
  695. -161> 2014-06-02 11:16:18.257924 7f02026f0700 5 -- op tracker -- , seq: 150812, time: 2014-06-02 11:16:18.257924, event: done, request: pg_notify(8.127(13) epoch 11116) v5
  696. -160> 2014-06-02 11:16:18.257930 7f02026f0700 10 osd.123 11123 do_waiters -- start
  697. -159> 2014-06-02 11:16:18.257932 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  698. -158> 2014-06-02 11:16:18.257937 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 48 ==== pg_notify(9.2fe(550) epoch 11116) v5 ==== 28872+0+0 (166888349 0 0) 0xe6340c0 con 0x10590000
  699. -157> 2014-06-02 11:16:18.257944 7f02026f0700 10 osd.123 11123 do_waiters -- start
  700. -156> 2014-06-02 11:16:18.257946 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  701. -155> 2014-06-02 11:16:18.257947 7f02026f0700 20 osd.123 11123 _dispatch 0xe6340c0 pg_notify(9.2fe(550) epoch 11116) v5
  702. -154> 2014-06-02 11:16:18.257950 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:06.984533, event: header_read, request: pg_notify(9.2fe(550) epoch 11116) v5
  703. -153> 2014-06-02 11:16:18.257954 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:06.984536, event: throttled, request: pg_notify(9.2fe(550) epoch 11116) v5
  704. -152> 2014-06-02 11:16:18.257957 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:06.985317, event: all_read, request: pg_notify(9.2fe(550) epoch 11116) v5
  705. -151> 2014-06-02 11:16:18.257960 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:18.257944, event: dispatched, request: pg_notify(9.2fe(550) epoch 11116) v5
  706. -150> 2014-06-02 11:16:18.257966 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:18.257966, event: waiting_for_osdmap, request: pg_notify(9.2fe(550) epoch 11116) v5
  707. -149> 2014-06-02 11:16:18.257971 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  708. -148> 2014-06-02 11:16:18.257973 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0xe6340c0
  709. -147> 2014-06-02 11:16:18.257976 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(9.2fe(550) epoch 11116) v5
  710. -146> 2014-06-02 11:16:18.257978 7f02026f0700 5 -- op tracker -- , seq: 150813, time: 2014-06-02 11:16:18.257978, event: done, request: pg_notify(9.2fe(550) epoch 11116) v5
  711. -145> 2014-06-02 11:16:18.257984 7f02026f0700 10 osd.123 11123 do_waiters -- start
  712. -144> 2014-06-02 11:16:18.257986 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  713. -143> 2014-06-02 11:16:18.258065 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 33 ==== pg_notify(13.330(17) epoch 11115) v5 ==== 1633+0+0 (170078448 0 0) 0x90dfc40 con 0x10c75c20
  714. -142> 2014-06-02 11:16:18.258077 7f02026f0700 10 osd.123 11123 do_waiters -- start
  715. -141> 2014-06-02 11:16:18.258079 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  716. -140> 2014-06-02 11:16:18.258080 7f02026f0700 20 osd.123 11123 _dispatch 0x90dfc40 pg_notify(13.330(17) epoch 11115) v5
  717. -139> 2014-06-02 11:16:18.258084 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:05.912048, event: header_read, request: pg_notify(13.330(17) epoch 11115) v5
  718. -138> 2014-06-02 11:16:18.258088 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:05.912050, event: throttled, request: pg_notify(13.330(17) epoch 11115) v5
  719. -137> 2014-06-02 11:16:18.258092 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:05.912166, event: all_read, request: pg_notify(13.330(17) epoch 11115) v5
  720. -136> 2014-06-02 11:16:18.258095 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:18.258077, event: dispatched, request: pg_notify(13.330(17) epoch 11115) v5
  721. -135> 2014-06-02 11:16:18.258099 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:18.258099, event: waiting_for_osdmap, request: pg_notify(13.330(17) epoch 11115) v5
  722. -134> 2014-06-02 11:16:18.258105 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  723. -133> 2014-06-02 11:16:18.258108 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x90dfc40
  724. -132> 2014-06-02 11:16:18.258110 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(13.330(17) epoch 11115) v5
  725. -131> 2014-06-02 11:16:18.258114 7f02026f0700 5 -- op tracker -- , seq: 150814, time: 2014-06-02 11:16:18.258113, event: done, request: pg_notify(13.330(17) epoch 11115) v5
  726. -130> 2014-06-02 11:16:18.258132 7f02026f0700 10 osd.123 11123 do_waiters -- start
  727. -129> 2014-06-02 11:16:18.258134 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  728. -128> 2014-06-02 11:16:18.258139 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.106 192.168.1.107:6802/13018623 74 ==== pg_notify(6.1bf(9) epoch 11116) v5 ==== 1209+0+0 (1963000356 0 0) 0x84a47c0 con 0x10589a20
  729. -127> 2014-06-02 11:16:18.258147 7f02026f0700 10 osd.123 11123 do_waiters -- start
  730. -126> 2014-06-02 11:16:18.258148 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  731. -125> 2014-06-02 11:16:18.258149 7f02026f0700 20 osd.123 11123 _dispatch 0x84a47c0 pg_notify(6.1bf(9) epoch 11116) v5
  732. -124> 2014-06-02 11:16:18.258156 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:07.004018, event: header_read, request: pg_notify(6.1bf(9) epoch 11116) v5
  733. -123> 2014-06-02 11:16:18.258160 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:07.004021, event: throttled, request: pg_notify(6.1bf(9) epoch 11116) v5
  734. -122> 2014-06-02 11:16:18.258164 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:07.004132, event: all_read, request: pg_notify(6.1bf(9) epoch 11116) v5
  735. -121> 2014-06-02 11:16:18.258169 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:18.258146, event: dispatched, request: pg_notify(6.1bf(9) epoch 11116) v5
  736. -120> 2014-06-02 11:16:18.258175 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:18.258175, event: waiting_for_osdmap, request: pg_notify(6.1bf(9) epoch 11116) v5
  737. -119> 2014-06-02 11:16:18.258180 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.106
  738. -118> 2014-06-02 11:16:18.258183 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x84a47c0
  739. -117> 2014-06-02 11:16:18.258185 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1bf(9) epoch 11116) v5
  740. -116> 2014-06-02 11:16:18.258188 7f02026f0700 5 -- op tracker -- , seq: 150815, time: 2014-06-02 11:16:18.258188, event: done, request: pg_notify(6.1bf(9) epoch 11116) v5
  741. -115> 2014-06-02 11:16:18.258194 7f02026f0700 10 osd.123 11123 do_waiters -- start
  742. -114> 2014-06-02 11:16:18.258196 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  743. -113> 2014-06-02 11:16:18.258200 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.9 192.168.1.103:6800/101063086 49 ==== pg_notify(6.1b8(14) epoch 11116) v5 ==== 1480+0+0 (2048473602 0 0) 0x64881c0 con 0x10590000
  744. -112> 2014-06-02 11:16:18.258208 7f02026f0700 10 osd.123 11123 do_waiters -- start
  745. -111> 2014-06-02 11:16:18.258210 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  746. -110> 2014-06-02 11:16:18.258211 7f02026f0700 20 osd.123 11123 _dispatch 0x64881c0 pg_notify(6.1b8(14) epoch 11116) v5
  747. -109> 2014-06-02 11:16:18.258214 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:07.002006, event: header_read, request: pg_notify(6.1b8(14) epoch 11116) v5
  748. -108> 2014-06-02 11:16:18.258218 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:07.002008, event: throttled, request: pg_notify(6.1b8(14) epoch 11116) v5
  749. -107> 2014-06-02 11:16:18.258221 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:07.002101, event: all_read, request: pg_notify(6.1b8(14) epoch 11116) v5
  750. -106> 2014-06-02 11:16:18.258226 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:18.258208, event: dispatched, request: pg_notify(6.1b8(14) epoch 11116) v5
  751. -105> 2014-06-02 11:16:18.258231 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:18.258231, event: waiting_for_osdmap, request: pg_notify(6.1b8(14) epoch 11116) v5
  752. -104> 2014-06-02 11:16:18.258237 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.9
  753. -103> 2014-06-02 11:16:18.258241 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x64881c0
  754. -102> 2014-06-02 11:16:18.258243 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(6.1b8(14) epoch 11116) v5
  755. -101> 2014-06-02 11:16:18.258246 7f02026f0700 5 -- op tracker -- , seq: 150816, time: 2014-06-02 11:16:18.258246, event: done, request: pg_notify(6.1b8(14) epoch 11116) v5
  756. -100> 2014-06-02 11:16:18.258252 7f02026f0700 10 osd.123 11123 do_waiters -- start
  757. -99> 2014-06-02 11:16:18.258254 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  758. -98> 2014-06-02 11:16:18.258259 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 34 ==== pg_notify(15.207(617) epoch 11115) v5 ==== 32585+0+0 (1695938112 0 0) 0x90da680 con 0x10c75c20
  759. -97> 2014-06-02 11:16:18.258266 7f02026f0700 10 osd.123 11123 do_waiters -- start
  760. -96> 2014-06-02 11:16:18.258268 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  761. -95> 2014-06-02 11:16:18.258269 7f02026f0700 20 osd.123 11123 _dispatch 0x90da680 pg_notify(15.207(617) epoch 11115) v5
  762. -94> 2014-06-02 11:16:18.258272 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:05.950326, event: header_read, request: pg_notify(15.207(617) epoch 11115) v5
  763. -93> 2014-06-02 11:16:18.258276 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:05.950329, event: throttled, request: pg_notify(15.207(617) epoch 11115) v5
  764. -92> 2014-06-02 11:16:18.258279 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:05.951213, event: all_read, request: pg_notify(15.207(617) epoch 11115) v5
  765. -91> 2014-06-02 11:16:18.258282 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:18.258266, event: dispatched, request: pg_notify(15.207(617) epoch 11115) v5
  766. -90> 2014-06-02 11:16:18.258287 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:18.258287, event: waiting_for_osdmap, request: pg_notify(15.207(617) epoch 11115) v5
  767. -89> 2014-06-02 11:16:18.258293 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  768. -88> 2014-06-02 11:16:18.258297 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x90da680
  769. -87> 2014-06-02 11:16:18.258299 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.207(617) epoch 11115) v5
  770. -86> 2014-06-02 11:16:18.258302 7f02026f0700 5 -- op tracker -- , seq: 150817, time: 2014-06-02 11:16:18.258302, event: done, request: pg_notify(15.207(617) epoch 11115) v5
  771. -85> 2014-06-02 11:16:18.258310 7f02026f0700 10 osd.123 11123 do_waiters -- start
  772. -84> 2014-06-02 11:16:18.258312 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  773. -83> 2014-06-02 11:16:18.258385 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 35 ==== pg_notify(3.1b7(15) epoch 11115) v5 ==== 1515+0+0 (2759906856 0 0) 0x193163c0 con 0x10c75c20
  774. -82> 2014-06-02 11:16:18.258397 7f02026f0700 10 osd.123 11123 do_waiters -- start
  775. -81> 2014-06-02 11:16:18.258399 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  776. -80> 2014-06-02 11:16:18.258400 7f02026f0700 20 osd.123 11123 _dispatch 0x193163c0 pg_notify(3.1b7(15) epoch 11115) v5
  777. -79> 2014-06-02 11:16:18.258403 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:05.988593, event: header_read, request: pg_notify(3.1b7(15) epoch 11115) v5
  778. -78> 2014-06-02 11:16:18.258408 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:05.988595, event: throttled, request: pg_notify(3.1b7(15) epoch 11115) v5
  779. -77> 2014-06-02 11:16:18.258411 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:05.988709, event: all_read, request: pg_notify(3.1b7(15) epoch 11115) v5
  780. -76> 2014-06-02 11:16:18.258414 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:18.258396, event: dispatched, request: pg_notify(3.1b7(15) epoch 11115) v5
  781. -75> 2014-06-02 11:16:18.258418 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:18.258418, event: waiting_for_osdmap, request: pg_notify(3.1b7(15) epoch 11115) v5
  782. -74> 2014-06-02 11:16:18.258423 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  783. -73> 2014-06-02 11:16:18.258426 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11115 (i am 11123) 0x193163c0
  784. -72> 2014-06-02 11:16:18.258429 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(15) epoch 11115) v5
  785. -71> 2014-06-02 11:16:18.258432 7f02026f0700 5 -- op tracker -- , seq: 150818, time: 2014-06-02 11:16:18.258431, event: done, request: pg_notify(3.1b7(15) epoch 11115) v5
  786. -70> 2014-06-02 11:16:18.258438 7f02026f0700 10 osd.123 11123 do_waiters -- start
  787. -69> 2014-06-02 11:16:18.258440 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  788. -68> 2014-06-02 11:16:18.258444 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 36 ==== osd_map(11116..11116 src has 3254..11116) v3 ==== 779+0+0 (1818745379 0 0) 0x5a2bcc0 con 0x10c75c20
  789. -67> 2014-06-02 11:16:18.258452 7f02026f0700 10 osd.123 11123 do_waiters -- start
  790. -66> 2014-06-02 11:16:18.258456 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  791. -65> 2014-06-02 11:16:18.258458 7f02026f0700 20 osd.123 11123 _dispatch 0x5a2bcc0 osd_map(11116..11116 src has 3254..11116) v3
  792. -64> 2014-06-02 11:16:18.258462 7f02026f0700 3 osd.123 11123 handle_osd_map epochs [11116,11116], i have 11123, src has [3254,11116]
  793. -63> 2014-06-02 11:16:18.258465 7f02026f0700 10 osd.123 11123 no new maps here, dropping
  794. -62> 2014-06-02 11:16:18.258466 7f02026f0700 10 osd.123 11123 do_waiters -- start
  795. -61> 2014-06-02 11:16:18.258467 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  796. -60> 2014-06-02 11:16:18.258493 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 37 ==== pg_notify(13.330(17) epoch 11116) v5 ==== 1633+0+0 (3738325713 0 0) 0x1a343800 con 0x10c75c20
  797. -59> 2014-06-02 11:16:18.258503 7f02026f0700 10 osd.123 11123 do_waiters -- start
  798. -58> 2014-06-02 11:16:18.258504 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  799. -57> 2014-06-02 11:16:18.258506 7f02026f0700 20 osd.123 11123 _dispatch 0x1a343800 pg_notify(13.330(17) epoch 11116) v5
  800. -56> 2014-06-02 11:16:18.258509 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:06.970631, event: header_read, request: pg_notify(13.330(17) epoch 11116) v5
  801. -55> 2014-06-02 11:16:18.258513 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:06.970632, event: throttled, request: pg_notify(13.330(17) epoch 11116) v5
  802. -54> 2014-06-02 11:16:18.258516 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:06.970717, event: all_read, request: pg_notify(13.330(17) epoch 11116) v5
  803. -53> 2014-06-02 11:16:18.258519 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:18.258503, event: dispatched, request: pg_notify(13.330(17) epoch 11116) v5
  804. -52> 2014-06-02 11:16:18.258523 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:18.258523, event: waiting_for_osdmap, request: pg_notify(13.330(17) epoch 11116) v5
  805. -51> 2014-06-02 11:16:18.258528 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  806. -50> 2014-06-02 11:16:18.258530 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x1a343800
  807. -49> 2014-06-02 11:16:18.258532 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(13.330(17) epoch 11116) v5
  808. -48> 2014-06-02 11:16:18.258535 7f02026f0700 5 -- op tracker -- , seq: 150819, time: 2014-06-02 11:16:18.258535, event: done, request: pg_notify(13.330(17) epoch 11116) v5
  809. -47> 2014-06-02 11:16:18.258541 7f02026f0700 10 osd.123 11123 do_waiters -- start
  810. -46> 2014-06-02 11:16:18.258543 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  811. -45> 2014-06-02 11:16:18.258548 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 38 ==== pg_notify(15.207(617) epoch 11116) v5 ==== 32585+0+0 (3656500016 0 0) 0x648a4c0 con 0x10c75c20
  812. -44> 2014-06-02 11:16:18.258555 7f02026f0700 10 osd.123 11123 do_waiters -- start
  813. -43> 2014-06-02 11:16:18.258557 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  814. -42> 2014-06-02 11:16:18.258558 7f02026f0700 20 osd.123 11123 _dispatch 0x648a4c0 pg_notify(15.207(617) epoch 11116) v5
  815. -41> 2014-06-02 11:16:18.258561 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:07.011018, event: header_read, request: pg_notify(15.207(617) epoch 11116) v5
  816. -40> 2014-06-02 11:16:18.258565 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:07.011020, event: throttled, request: pg_notify(15.207(617) epoch 11116) v5
  817. -39> 2014-06-02 11:16:18.258569 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:07.011886, event: all_read, request: pg_notify(15.207(617) epoch 11116) v5
  818. -38> 2014-06-02 11:16:18.258572 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:18.258555, event: dispatched, request: pg_notify(15.207(617) epoch 11116) v5
  819. -37> 2014-06-02 11:16:18.258577 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:18.258577, event: waiting_for_osdmap, request: pg_notify(15.207(617) epoch 11116) v5
  820. -36> 2014-06-02 11:16:18.258582 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  821. -35> 2014-06-02 11:16:18.258584 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x648a4c0
  822. -34> 2014-06-02 11:16:18.258586 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(15.207(617) epoch 11116) v5
  823. -33> 2014-06-02 11:16:18.258589 7f02026f0700 5 -- op tracker -- , seq: 150820, time: 2014-06-02 11:16:18.258589, event: done, request: pg_notify(15.207(617) epoch 11116) v5
  824. -32> 2014-06-02 11:16:18.258595 7f02026f0700 10 osd.123 11123 do_waiters -- start
  825. -31> 2014-06-02 11:16:18.258596 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  826. -30> 2014-06-02 11:16:18.258774 7f02026f0700 1 -- 192.168.1.114:6800/17052303 <== osd.108 192.168.1.108:6838/82013669 39 ==== pg_notify(3.1b7(15) epoch 11116) v5 ==== 1515+0+0 (3448872760 0 0) 0x6488e00 con 0x10c75c20
  827. -29> 2014-06-02 11:16:18.258786 7f02026f0700 10 osd.123 11123 do_waiters -- start
  828. -28> 2014-06-02 11:16:18.258788 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  829. -27> 2014-06-02 11:16:18.258790 7f02026f0700 20 osd.123 11123 _dispatch 0x6488e00 pg_notify(3.1b7(15) epoch 11116) v5
  830. -26> 2014-06-02 11:16:18.258793 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:07.052508, event: header_read, request: pg_notify(3.1b7(15) epoch 11116) v5
  831. -25> 2014-06-02 11:16:18.258797 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:07.052509, event: throttled, request: pg_notify(3.1b7(15) epoch 11116) v5
  832. -24> 2014-06-02 11:16:18.258801 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:07.052570, event: all_read, request: pg_notify(3.1b7(15) epoch 11116) v5
  833. -23> 2014-06-02 11:16:18.258804 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:18.258786, event: dispatched, request: pg_notify(3.1b7(15) epoch 11116) v5
  834. -22> 2014-06-02 11:16:18.258809 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:18.258809, event: waiting_for_osdmap, request: pg_notify(3.1b7(15) epoch 11116) v5
  835. -21> 2014-06-02 11:16:18.258815 7f02026f0700 7 osd.123 11123 handle_pg_notify from osd.108
  836. -20> 2014-06-02 11:16:18.258818 7f02026f0700 15 osd.123 11123 require_same_or_newer_map 11116 (i am 11123) 0x6488e00
  837. -19> 2014-06-02 11:16:18.258820 7f02026f0700 7 osd.123 11123 still in boot state, dropping message pg_notify(3.1b7(15) epoch 11116) v5
  838. -18> 2014-06-02 11:16:18.258823 7f02026f0700 5 -- op tracker -- , seq: 150821, time: 2014-06-02 11:16:18.258823, event: done, request: pg_notify(3.1b7(15) epoch 11116) v5
  839. -17> 2014-06-02 11:16:18.258830 7f02026f0700 10 osd.123 11123 do_waiters -- start
  840. -16> 2014-06-02 11:16:18.258832 7f02026f0700 10 osd.123 11123 do_waiters -- finish
  841. -15> 2014-06-02 11:16:18.752279 7f020f2b5700 5 osd.123 11123 tick
  842. -14> 2014-06-02 11:16:18.752305 7f020f2b5700 10 osd.123 11123 maybe_update_heartbeat_peers updating
  843. -13> 2014-06-02 11:16:18.752311 7f020f2b5700 10 osd.123 11123 adding neighbor peer osd.120
  844. -12> 2014-06-02 11:16:18.752315 7f020f2b5700 10 osd.123 11123 adding neighbor peer osd.124
  845. -11> 2014-06-02 11:16:18.752320 7f020f2b5700 10 osd.123 11123 maybe_update_heartbeat_peers 10 peers, extras 120,124
  846. -10> 2014-06-02 11:16:18.752333 7f020f2b5700 1 osd.123 11123 is_healthy false -- only 0/10 up peers (less than 1/3)
  847. -9> 2014-06-02 11:16:18.752337 7f020f2b5700 10 osd.123 11123 do_waiters -- start
  848. -8> 2014-06-02 11:16:18.752339 7f020f2b5700 10 osd.123 11123 do_waiters -- finish
  849. -7> 2014-06-02 11:16:19.752413 7f020f2b5700 5 osd.123 11123 tick
  850. -6> 2014-06-02 11:16:19.752450 7f020f2b5700 1 osd.123 11123 is_healthy false -- only 0/10 up peers (less than 1/3)
  851. -5> 2014-06-02 11:16:19.752455 7f020f2b5700 10 osd.123 11123 do_waiters -- start
  852. -4> 2014-06-02 11:16:19.752457 7f020f2b5700 10 osd.123 11123 do_waiters -- finish
  853. -3> 2014-06-02 11:16:19.845896 7f01f13ac700 2 -- 192.168.1.114:0/52303 >> 192.168.100.115:6842/12029171 pipe(0x16fdf080 sd=191 :52000 s=1 pgs=0 cs=0 l=1 c=0xbf0ce60). got newly_acked_seq 0 vs out_seq 0
  854. -2> 2014-06-02 11:16:19.846041 7f01f14ad700 2 -- 192.168.1.114:0/52303 >> 192.168.1.115:6842/12029171 pipe(0x16fdc600 sd=167 :47693 s=1 pgs=0 cs=0 l=1 c=0x540fe80). got newly_acked_seq 0 vs out_seq 0
  855. -1> 2014-06-02 11:16:19.859381 7f01f34cd700 2 -- 192.168.1.114:6800/17052303 >> 192.168.1.115:6802/73028675 pipe(0x1787af80 sd=166 :59845 s=1 pgs=0 cs=0 l=0 c=0x10e353e0). got newly_acked_seq 64 vs out_seq 0
  856. 0> 2014-06-02 11:16:19.861410 7f01f34cd700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f01f34cd700 time 2014-06-02 11:16:19.859433
  857. msg/Pipe.cc: 1070: FAILED assert(m)
  858.  
  859. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  860. 1: (Pipe::connect()+0x3d0c) [0xb2931c]
  861. 2: (Pipe::writer()+0x9f3) [0xb2a063]
  862. 3: (Pipe::Writer::entry()+0xd) [0xb359cd]
  863. 4: (()+0x7851) [0x7f02155fc851]
  864. 5: (clone()+0x6d) [0x7f021433894d]
  865. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  866.  
  867. --- logging levels ---
  868. 0/ 5 none
  869. 0/ 1 lockdep
  870. 0/ 1 context
  871. 1/ 1 crush
  872. 1/ 5 mds
  873. 1/ 5 mds_balancer
  874. 1/ 5 mds_locker
  875. 1/ 5 mds_log
  876. 1/ 5 mds_log_expire
  877. 1/ 5 mds_migrator
  878. 0/ 1 buffer
  879. 0/ 1 timer
  880. 0/ 1 filer
  881. 0/ 1 striper
  882. 0/ 1 objecter
  883. 0/ 5 rados
  884. 0/ 5 rbd
  885. 0/ 5 journaler
  886. 0/ 5 objectcacher
  887. 0/ 5 client
  888. 20/20 osd
  889. 0/ 5 optracker
  890. 0/ 5 objclass
  891. 1/ 3 filestore
  892. 1/ 3 keyvaluestore
  893. 1/ 3 journal
  894. 0/ 5 ms
  895. 1/ 5 mon
  896. 0/10 monc
  897. 1/ 5 paxos
  898. 0/ 5 tp
  899. 1/ 5 auth
  900. 1/ 5 crypto
  901. 1/ 1 finisher
  902. 1/ 5 heartbeatmap
  903. 1/ 5 perfcounter
  904. 1/ 5 rgw
  905. 1/ 5 javaclient
  906. 1/ 5 asok
  907. 1/ 1 throttle
  908. -2/-2 (syslog threshold)
  909. -1/-1 (stderr threshold)
  910. max_recent 10000
  911. max_new 1000
  912. log_file /ceph/storage0114/log/ceph/ceph-osd.123.log
  913. --- end dump of recent events ---
  914. 2014-06-02 11:16:19.969707 7f01f34cd700 -1 *** Caught signal (Aborted) **
  915. in thread 7f01f34cd700
  916.  
  917. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  918. 1: /usr/bin/ceph-osd() [0x9ab3b1]
  919. 2: (()+0xf500) [0x7f0215604500]
  920. 3: (gsignal()+0x35) [0x7f02142828e5]
  921. 4: (abort()+0x175) [0x7f02142840c5]
  922. 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f0214b3ba5d]
  923. 6: (()+0xbcbe6) [0x7f0214b39be6]
  924. 7: (()+0xbcc13) [0x7f0214b39c13]
  925. 8: (()+0xbcd0e) [0x7f0214b39d0e]
  926. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f2) [0xaec612]
  927. 10: (Pipe::connect()+0x3d0c) [0xb2931c]
  928. 11: (Pipe::writer()+0x9f3) [0xb2a063]
  929. 12: (Pipe::Writer::entry()+0xd) [0xb359cd]
  930. 13: (()+0x7851) [0x7f02155fc851]
  931. 14: (clone()+0x6d) [0x7f021433894d]
  932. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  933.  
  934. --- begin dump of recent events ---
  935. 0> 2014-06-02 11:16:19.969707 7f01f34cd700 -1 *** Caught signal (Aborted) **
  936. in thread 7f01f34cd700
  937.  
  938. ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  939. 1: /usr/bin/ceph-osd() [0x9ab3b1]
  940. 2: (()+0xf500) [0x7f0215604500]
  941. 3: (gsignal()+0x35) [0x7f02142828e5]
  942. 4: (abort()+0x175) [0x7f02142840c5]
  943. 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f0214b3ba5d]
  944. 6: (()+0xbcbe6) [0x7f0214b39be6]
  945. 7: (()+0xbcc13) [0x7f0214b39c13]
  946. 8: (()+0xbcd0e) [0x7f0214b39d0e]
  947. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f2) [0xaec612]
  948. 10: (Pipe::connect()+0x3d0c) [0xb2931c]
  949. 11: (Pipe::writer()+0x9f3) [0xb2a063]
  950. 12: (Pipe::Writer::entry()+0xd) [0xb359cd]
  951. 13: (()+0x7851) [0x7f02155fc851]
  952. 14: (clone()+0x6d) [0x7f021433894d]
  953. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  954.  
  955. --- logging levels ---
  956. 0/ 5 none
  957. 0/ 1 lockdep
  958. 0/ 1 context
  959. 1/ 1 crush
  960. 1/ 5 mds
  961. 1/ 5 mds_balancer
  962. 1/ 5 mds_locker
  963. 1/ 5 mds_log
  964. 1/ 5 mds_log_expire
  965. 1/ 5 mds_migrator
  966. 0/ 1 buffer
  967. 0/ 1 timer
  968. 0/ 1 filer
  969. 0/ 1 striper
  970. 0/ 1 objecter
  971. 0/ 5 rados
  972. 0/ 5 rbd
  973. 0/ 5 journaler
  974. 0/ 5 objectcacher
  975. 0/ 5 client
  976. 20/20 osd
  977. 0/ 5 optracker
  978. 0/ 5 objclass
  979. 1/ 3 filestore
  980. 1/ 3 keyvaluestore
  981. 1/ 3 journal
  982. 0/ 5 ms
  983. 1/ 5 mon
  984. 0/10 monc
  985. 1/ 5 paxos
  986. 0/ 5 tp
  987. 1/ 5 auth
  988. 1/ 5 crypto
  989. 1/ 1 finisher
  990. 1/ 5 heartbeatmap
  991. 1/ 5 perfcounter
  992. 1/ 5 rgw
  993. 1/ 5 javaclient
  994. 1/ 5 asok
  995. 1/ 1 throttle
  996. -2/-2 (syslog threshold)
  997. -1/-1 (stderr threshold)
  998. max_recent 10000
  999. max_new 1000
  1000. log_file /ceph/storage0114/log/ceph/ceph-osd.123.log
  1001. --- end dump of recent events ---
  1002. [root@storage0114-ib ceph]#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!