API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jun 4th, 2013
132
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 41.04 KB | None | 0 0
raw download clone embed print report
  1. root@Ceph1:~# ceph-mds -i 1 -d
  2. 2013-06-04 18:07:46.983018 7fadd60617c0 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-mds, pid 11826
  3. starting mds.1 at :/0
  4. 2013-06-04 18:07:47.387643 7fadd0aa6700 1 mds.-1.0 handle_mds_map standby
  5. 2013-06-04 18:07:47.704154 7fadd0aa6700 1 mds.0.45 handle_mds_map i am now mds.0.45
  6. 2013-06-04 18:07:47.704159 7fadd0aa6700 1 mds.0.45 handle_mds_map state change up:standby --> up:replay
  7. 2013-06-04 18:07:47.704171 7fadd0aa6700 1 mds.0.45 replay_start
  8. 2013-06-04 18:07:47.704190 7fadd0aa6700 1 mds.0.45 recovery set is
  9. 2013-06-04 18:07:47.704193 7fadd0aa6700 1 mds.0.45 need osdmap epoch 1710, have 1709
  10. 2013-06-04 18:07:47.704196 7fadd0aa6700 1 mds.0.45 waiting for osdmap 1710 (which blacklists prior instance)
  11. 2013-06-04 18:07:47.711616 7fadd0aa6700 0 mds.0.cache creating system inode with ino:100
  12. 2013-06-04 18:07:47.712032 7fadd0aa6700 0 mds.0.cache creating system inode with ino:1
  13. mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7fadcd594700 time 2013-06-04 18:07:49.260303
  14. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  15. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  16. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  17. 2: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  18. 3: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  19. 4: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  20. 5: (()+0x7f8e) [0x7fadd5c3ef8e]
  21. 6: (clone()+0x6d) [0x7fadd455be1d]
  22. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  23. 2013-06-04 18:07:49.261299 7fadcd594700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7fadcd594700 time 2013-06-04 18:07:49.260303
  24. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  25.  
  26. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  27. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  28. 2: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  29. 3: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  30. 4: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  31. 5: (()+0x7f8e) [0x7fadd5c3ef8e]
  32. 6: (clone()+0x6d) [0x7fadd455be1d]
  33. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  34.  
  35. --- begin dump of recent events ---
  36. -205> 2013-06-04 18:07:46.979593 7fadd60617c0 5 asok(0x1c9a000) register_command perfcounters_dump hook 0x1c84010
  37. -204> 2013-06-04 18:07:46.979664 7fadd60617c0 5 asok(0x1c9a000) register_command 1 hook 0x1c84010
  38. -203> 2013-06-04 18:07:46.979669 7fadd60617c0 5 asok(0x1c9a000) register_command perf dump hook 0x1c84010
  39. -202> 2013-06-04 18:07:46.979684 7fadd60617c0 5 asok(0x1c9a000) register_command perfcounters_schema hook 0x1c84010
  40. -201> 2013-06-04 18:07:46.979692 7fadd60617c0 5 asok(0x1c9a000) register_command 2 hook 0x1c84010
  41. -200> 2013-06-04 18:07:46.979695 7fadd60617c0 5 asok(0x1c9a000) register_command perf schema hook 0x1c84010
  42. -199> 2013-06-04 18:07:46.979701 7fadd60617c0 5 asok(0x1c9a000) register_command config show hook 0x1c84010
  43. -198> 2013-06-04 18:07:46.979706 7fadd60617c0 5 asok(0x1c9a000) register_command config set hook 0x1c84010
  44. -197> 2013-06-04 18:07:46.979720 7fadd60617c0 5 asok(0x1c9a000) register_command log flush hook 0x1c84010
  45. -196> 2013-06-04 18:07:46.979725 7fadd60617c0 5 asok(0x1c9a000) register_command log dump hook 0x1c84010
  46. -195> 2013-06-04 18:07:46.979729 7fadd60617c0 5 asok(0x1c9a000) register_command log reopen hook 0x1c84010
  47. -194> 2013-06-04 18:07:46.983018 7fadd60617c0 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-mds, pid 11826
  48. -193> 2013-06-04 18:07:46.983659 7fadd60617c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/11826 need_addr=1
  49. -192> 2013-06-04 18:07:46.987420 7fadd60617c0 5 asok(0x1c9a000) init /var/run/ceph/ceph-mds.1.asok
  50. -191> 2013-06-04 18:07:46.987463 7fadd60617c0 5 asok(0x1c9a000) bind_and_listen /var/run/ceph/ceph-mds.1.asok
  51. -190> 2013-06-04 18:07:46.987541 7fadd60617c0 5 asok(0x1c9a000) register_command 0 hook 0x1c820b8
  52. -189> 2013-06-04 18:07:46.987562 7fadd60617c0 5 asok(0x1c9a000) register_command version hook 0x1c820b8
  53. -188> 2013-06-04 18:07:46.987569 7fadd60617c0 5 asok(0x1c9a000) register_command git_version hook 0x1c820b8
  54. -187> 2013-06-04 18:07:46.987575 7fadd60617c0 5 asok(0x1c9a000) register_command help hook 0x1c840d0
  55. -186> 2013-06-04 18:07:46.987650 7fadd60617c0 10 monclient(hunting): build_initial_monmap
  56. -185> 2013-06-04 18:07:46.987677 7fadd22a9700 5 asok(0x1c9a000) entry start
  57. -184> 2013-06-04 18:07:46.987763 7fadd60617c0 1 -- 0.0.0.0:6800/11826 messenger.start
  58. -183> 2013-06-04 18:07:46.987946 7fadd60617c0 5 adding auth protocol: cephx
  59. -182> 2013-06-04 18:07:46.987962 7fadd60617c0 5 adding auth protocol: cephx
  60. -181> 2013-06-04 18:07:46.988132 7fadd60617c0 1 accepter.accepter.start
  61. -180> 2013-06-04 18:07:46.988181 7fadd60617c0 10 monclient(hunting): init
  62. -179> 2013-06-04 18:07:46.988188 7fadd60617c0 5 adding auth protocol: cephx
  63. -178> 2013-06-04 18:07:46.988200 7fadd60617c0 10 monclient(hunting): auth_supported 2 method cephx
  64. -177> 2013-06-04 18:07:46.988506 7fadd60617c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/keyring.mds.1
  65. -176> 2013-06-04 18:07:46.988615 7fadd60617c0 10 monclient(hunting): _reopen_session
  66. -175> 2013-06-04 18:07:46.988708 7fadd60617c0 10 monclient(hunting): _pick_new_mon picked mon.2 con 0x1cacdc0 addr 10.10.4.232:6789/0
  67. -174> 2013-06-04 18:07:46.988744 7fadd60617c0 10 monclient(hunting): _send_mon_message to mon.2 at 10.10.4.232:6789/0
  68. -173> 2013-06-04 18:07:46.988752 7fadd60617c0 1 -- 0.0.0.0:6800/11826 --> 10.10.4.232:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1c986c0 con 0x1cacdc0
  69. -172> 2013-06-04 18:07:46.988786 7fadd60617c0 10 monclient(hunting): renew_subs
  70. -171> 2013-06-04 18:07:46.989700 7fadd605f700 1 -- 10.10.4.231:6800/11826 learned my addr 10.10.4.231:6800/11826
  71. -170> 2013-06-04 18:07:46.990161 7fadd0aa6700 5 mds.-1.0 ms_handle_connect on 10.10.4.232:6789/0
  72. -169> 2013-06-04 18:07:46.991028 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 1 ==== mon_map v1 ==== 473+0+0 (3706936977 0 0) 0x1cec3c0 con 0x1cacdc0
  73. -168> 2013-06-04 18:07:46.991062 7fadd0aa6700 10 monclient(hunting): handle_monmap mon_map v1
  74. -167> 2013-06-04 18:07:46.991089 7fadd0aa6700 10 monclient(hunting): got monmap 1, mon.2 is now rank 1
  75. -166> 2013-06-04 18:07:46.991093 7fadd0aa6700 10 monclient(hunting): dump:
  76. epoch 1
  77. fsid 98f5178a-6c39-4fcf-8ebe-4250c09a8b69
  78. last_changed 2013-04-25 16:28:37.785104
  79. created 2013-04-25 16:28:37.785104
  80. 0: 10.10.4.231:6789/0 mon.1
  81. 1: 10.10.4.232:6789/0 mon.2
  82. 2: 10.10.4.233:6789/0 mon.3
  83.  
  84. -165> 2013-06-04 18:07:46.991149 7fadd0aa6700 1 monclient(hunting): found mon.2
  85. -164> 2013-06-04 18:07:46.991167 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (434806622 0 0) 0x1ca9400 con 0x1cacdc0
  86. -163> 2013-06-04 18:07:46.991213 7fadd0aa6700 10 monclient: my global_id is 16633
  87. -162> 2013-06-04 18:07:46.991500 7fadd0aa6700 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  88. -161> 2013-06-04 18:07:46.991519 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1c98900 con 0x1cacdc0
  89. -160> 2013-06-04 18:07:46.992741 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (664972728 0 0) 0x1ca9800 con 0x1cacdc0
  90. -159> 2013-06-04 18:07:46.992930 7fadd0aa6700 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  91. -158> 2013-06-04 18:07:46.992947 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1c98d80 con 0x1cacdc0
  92. -157> 2013-06-04 18:07:46.994268 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (2031963219 0 0) 0x1ca9600 con 0x1cacdc0
  93. -156> 2013-06-04 18:07:46.994420 7fadd0aa6700 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  94. -155> 2013-06-04 18:07:46.994435 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1cca380 con 0x1cacdc0
  95. -154> 2013-06-04 18:07:46.994484 7fadd0aa6700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-06-04 18:07:16.994483)
  96. -153> 2013-06-04 18:07:46.994503 7fadd0aa6700 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  97. -152> 2013-06-04 18:07:46.994513 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1c98b40 con 0x1cacdc0
  98. -151> 2013-06-04 18:07:46.994593 7fadd60617c0 5 monclient: authenticate success, global_id 16633
  99. -150> 2013-06-04 18:07:46.994617 7fadd60617c0 10 monclient: wait_auth_rotating waiting (until 2013-06-04 18:08:16.994615)
  100. -149> 2013-06-04 18:07:46.995127 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 5 ==== mon_map v1 ==== 473+0+0 (3706936977 0 0) 0x1cec5a0 con 0x1cacdc0
  101. -148> 2013-06-04 18:07:46.995167 7fadd0aa6700 10 monclient: handle_monmap mon_map v1
  102. -147> 2013-06-04 18:07:46.995188 7fadd0aa6700 10 monclient: got monmap 1, mon.2 is now rank 1
  103. -146> 2013-06-04 18:07:46.995195 7fadd0aa6700 10 monclient: dump:
  104. epoch 1
  105. fsid 98f5178a-6c39-4fcf-8ebe-4250c09a8b69
  106. last_changed 2013-04-25 16:28:37.785104
  107. created 2013-04-25 16:28:37.785104
  108. 0: 10.10.4.231:6789/0 mon.1
  109. 1: 10.10.4.232:6789/0 mon.2
  110. 2: 10.10.4.233:6789/0 mon.3
  111.  
  112. -145> 2013-06-04 18:07:46.995236 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3723835308 0 0) 0x1cca540 con 0x1cacdc0
  113. -144> 2013-06-04 18:07:46.995273 7fadd0aa6700 10 monclient: handle_subscribe_ack sent 2013-06-04 18:07:46.988793 renew after 2013-06-04 18:10:16.988793
  114. -143> 2013-06-04 18:07:46.995321 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2417554023 0 0) 0x1ca9a00 con 0x1cacdc0
  115. -142> 2013-06-04 18:07:46.995441 7fadd0aa6700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-06-04 18:07:16.995441)
  116. -141> 2013-06-04 18:07:46.995520 7fadd60617c0 10 monclient: wait_auth_rotating done
  117. -140> 2013-06-04 18:07:46.995567 7fadd60617c0 5 asok(0x1c9a000) register_command objecter_requests hook 0x1c84100
  118. -139> 2013-06-04 18:07:46.995652 7fadd60617c0 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  119. -138> 2013-06-04 18:07:46.995665 7fadd60617c0 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- mdsbeacon(16633/1 up:boot seq 1 v0) v2 -- ?+0 0x1cd42c0 con 0x1cacdc0
  120. -137> 2013-06-04 18:07:46.995695 7fadd60617c0 10 monclient: renew_subs
  121. -136> 2013-06-04 18:07:46.995715 7fadd60617c0 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  122. -135> 2013-06-04 18:07:46.995721 7fadd60617c0 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x1cca1c0 con 0x1cacdc0
  123. -134> 2013-06-04 18:07:46.995735 7fadd60617c0 10 monclient: renew_subs
  124. -133> 2013-06-04 18:07:46.995740 7fadd60617c0 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  125. -132> 2013-06-04 18:07:46.995746 7fadd60617c0 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x1ccaa80 con 0x1cacdc0
  126. -131> 2013-06-04 18:07:46.996822 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 8 ==== osd_map(1709..1709 src has 1078..1709) v3 ==== 4824+0+0 (3496231919 0 0) 0x1c99200 con 0x1cacdc0
  127. -130> 2013-06-04 18:07:46.997077 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3723835308 0 0) 0x1cca1c0 con 0x1cacdc0
  128. -129> 2013-06-04 18:07:46.997101 7fadd0aa6700 10 monclient: handle_subscribe_ack sent 2013-06-04 18:07:46.995700 renew after 2013-06-04 18:10:16.995700
  129. -128> 2013-06-04 18:07:46.997122 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 10 ==== mdsmap(e 1411) v1 ==== 583+0+0 (3653044800 0 0) 0x1ca9800 con 0x1cacdc0
  130. -127> 2013-06-04 18:07:46.997142 7fadd0aa6700 5 mds.-1.0 handle_mds_map epoch 1411 from mon.1
  131. -126> 2013-06-04 18:07:46.997206 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 11 ==== osd_map(1709..1709 src has 1078..1709) v3 ==== 4824+0+0 (3496231919 0 0) 0x1c99680 con 0x1cacdc0
  132. -125> 2013-06-04 18:07:46.997236 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3723835308 0 0) 0x1cca380 con 0x1cacdc0
  133. -124> 2013-06-04 18:07:46.997251 7fadd0aa6700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  134. -123> 2013-06-04 18:07:47.387517 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 13 ==== mdsmap(e 1412) v1 ==== 575+0+0 (2297629642 0 0) 0x1ca9400 con 0x1cacdc0
  135. -122> 2013-06-04 18:07:47.387568 7fadd0aa6700 5 mds.-1.-1 handle_mds_map epoch 1412 from mon.1
  136. -121> 2013-06-04 18:07:47.387629 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 mark_down 10.10.4.231:6800/11005 -- pipe dne
  137. -120> 2013-06-04 18:07:47.387643 7fadd0aa6700 1 mds.-1.0 handle_mds_map standby
  138. -119> 2013-06-04 18:07:47.704057 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 14 ==== mdsmap(e 1413) v1 ==== 583+0+0 (3147310481 0 0) 0x1ca9c00 con 0x1cacdc0
  139. -118> 2013-06-04 18:07:47.704106 7fadd0aa6700 5 mds.-1.0 handle_mds_map epoch 1413 from mon.1
  140. -117> 2013-06-04 18:07:47.704154 7fadd0aa6700 1 mds.0.45 handle_mds_map i am now mds.0.45
  141. -116> 2013-06-04 18:07:47.704159 7fadd0aa6700 1 mds.0.45 handle_mds_map state change up:standby --> up:replay
  142. -115> 2013-06-04 18:07:47.704171 7fadd0aa6700 1 mds.0.45 replay_start
  143. -114> 2013-06-04 18:07:47.704190 7fadd0aa6700 1 mds.0.45 recovery set is
  144. -113> 2013-06-04 18:07:47.704193 7fadd0aa6700 1 mds.0.45 need osdmap epoch 1710, have 1709
  145. -112> 2013-06-04 18:07:47.704196 7fadd0aa6700 1 mds.0.45 waiting for osdmap 1710 (which blacklists prior instance)
  146. -111> 2013-06-04 18:07:47.704214 7fadd0aa6700 10 monclient: renew_subs
  147. -110> 2013-06-04 18:07:47.704222 7fadd0aa6700 10 monclient: _send_mon_message to mon.2 at 10.10.4.232:6789/0
  148. -109> 2013-06-04 18:07:47.704230 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6789/0 -- mon_subscribe({mdsmap=1414+,monmap=2+,osdmap=1710}) v2 -- ?+0 0x1cca380 con 0x1cacdc0
  149. -108> 2013-06-04 18:07:47.705104 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 15 ==== osd_map(1710..1710 src has 1078..1710) v3 ==== 308+0+0 (1194742878 0 0) 0x1c98fc0 con 0x1cacdc0
  150. -107> 2013-06-04 18:07:47.705212 7fadd0aa6700 2 mds.0.45 boot_start 1: opening inotable
  151. -106> 2013-06-04 18:07:47.705441 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:1 mds0_inotable [read 0~0] 1.b852b893 e1710) v4 -- ?+0 0x1c99440 con 0x1cade40
  152. -105> 2013-06-04 18:07:47.705517 7fadd0aa6700 2 mds.0.45 boot_start 1: opening sessionmap
  153. -104> 2013-06-04 18:07:47.705606 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6800/7185 -- osd_op(mds.0.45:2 mds0_sessionmap [read 0~0] 1.3270c60b e1710) v4 -- ?+0 0x1c99d40 con 0x1d04000
  154. -103> 2013-06-04 18:07:47.705631 7fadd0aa6700 2 mds.0.45 boot_start 1: opening anchor table
  155. -102> 2013-06-04 18:07:47.705653 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:3 mds_anchortable [read 0~0] 1.a977f6a7 e1710) v4 -- ?+0 0x1c99b00 con 0x1cade40
  156. -101> 2013-06-04 18:07:47.705667 7fadd0aa6700 2 mds.0.45 boot_start 1: opening snap table
  157. -100> 2013-06-04 18:07:47.705756 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.236:6800/2741 -- osd_op(mds.0.45:4 mds_snaptable [read 0~0] 1.d90270ad e1710) v4 -- ?+0 0x1c998c0 con 0x1d04420
  158. -99> 2013-06-04 18:07:47.705788 7fadd0aa6700 2 mds.0.45 boot_start 1: opening mds log
  159. -98> 2013-06-04 18:07:47.705792 7fadd0aa6700 5 mds.0.log open discovering log bounds
  160. -97> 2013-06-04 18:07:47.705802 7fadd0aa6700 1 mds.0.journaler(ro) recover start
  161. -96> 2013-06-04 18:07:47.705806 7fadd0aa6700 1 mds.0.journaler(ro) read_head
  162. -95> 2013-06-04 18:07:47.705890 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:5 200.00000000 [read 0~0] 1.844f3494 e1710) v4 -- ?+0 0x1c99680 con 0x1d04b00
  163. -94> 2013-06-04 18:07:47.705938 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== mon.1 10.10.4.232:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3723835308 0 0) 0x1ccac40 con 0x1cacdc0
  164. -93> 2013-06-04 18:07:47.705955 7fadd0aa6700 10 monclient: handle_subscribe_ack sent 2013-06-04 18:07:47.704218 renew after 2013-06-04 18:10:17.704218
  165. -92> 2013-06-04 18:07:47.707141 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.231:6801/8051
  166. -91> 2013-06-04 18:07:47.707172 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.232:6800/7185
  167. -90> 2013-06-04 18:07:47.707243 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.235:6800/4094
  168. -89> 2013-06-04 18:07:47.707257 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.236:6800/2741
  169. -88> 2013-06-04 18:07:47.708663 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 1 ==== osd_op_reply(5 200.00000000 [read 0~84] ondisk = 0) v4 ==== 111+0+84 (1046445087 0 1721677399) 0x1ca9e00 con 0x1d04b00
  170. -87> 2013-06-04 18:07:47.708743 7fadd0aa6700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 478150656, expire 478151490, write 605530472). probing for end of log (from 605530472)...
  171. -86> 2013-06-04 18:07:47.708762 7fadd0aa6700 1 mds.0.journaler(ro) probing for end of the log
  172. -85> 2013-06-04 18:07:47.708921 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:6 200.00000090 [stat] 1.13366537 rwordered e1710) v4 -- ?+0 0x1c98fc0 con 0x1d04840
  173. -84> 2013-06-04 18:07:47.708959 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:7 200.00000091 [stat] 1.4f213ac2 rwordered e1710) v4 -- ?+0 0x1c99200 con 0x1d04b00
  174. -83> 2013-06-04 18:07:47.709012 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.2 10.10.4.232:6800/7185 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~226] ondisk = 0) v4 ==== 114+0+226 (3563078536 0 3290022754) 0x1d20000 con 0x1d04000
  175. -82> 2013-06-04 18:07:47.709091 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.6 10.10.4.236:6800/2741 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (2107855555 0 2312406677) 0x1d22200 con 0x1d04420
  176. -81> 2013-06-04 18:07:47.709133 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 1 ==== osd_op_reply(3 mds_anchortable [read 0~29] ondisk = 0) v4 ==== 114+0+29 (2859428580 0 218791405) 0x1d22000 con 0x1cade40
  177. -80> 2013-06-04 18:07:47.709162 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 2 ==== osd_op_reply(1 mds0_inotable [read 0~34] ondisk = 0) v4 ==== 112+0+34 (1899832887 0 636700455) 0x1d22600 con 0x1cade40
  178. -79> 2013-06-04 18:07:47.709783 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 2 ==== osd_op_reply(7 200.00000091 [stat] ack = -2 (No such file or directory)) v4 ==== 111+0+0 (1698834434 0 0) 0x1ca9c00 con 0x1d04b00
  179. -78> 2013-06-04 18:07:47.710184 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.234:6800/6714
  180. -77> 2013-06-04 18:07:47.711517 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 1 ==== osd_op_reply(6 200.00000090 [stat] ondisk = 0) v4 ==== 111+0+16 (3130045244 0 3906419480) 0x1ca9e00 con 0x1d04840
  181. -76> 2013-06-04 18:07:47.711591 7fadd0aa6700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 605537636 (header had 605530472). recovered.
  182. -75> 2013-06-04 18:07:47.711604 7fadd0aa6700 2 mds.0.45 boot_start 2: loading/discovering base inodes
  183. -74> 2013-06-04 18:07:47.711616 7fadd0aa6700 0 mds.0.cache creating system inode with ino:100
  184. -73> 2013-06-04 18:07:47.711777 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:8 100.00000000 [getxattr inode] 1.c5265ab3 e1710) v4 -- ?+0 0x1d0f200 con 0x1d04b00
  185. -72> 2013-06-04 18:07:47.711844 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:9 100.00000000.inode [read 0~0] 1.85dde07f e1710) v4 -- ?+0 0x1d0efc0 con 0x1d04b00
  186. -71> 2013-06-04 18:07:47.712032 7fadd0aa6700 0 mds.0.cache creating system inode with ino:1
  187. -70> 2013-06-04 18:07:47.712185 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:10 1.00000000 [getxattr inode] 1.6b2cdaff e1710) v4 -- ?+0 0x1d0eb40 con 0x1d058c0
  188. -69> 2013-06-04 18:07:47.712236 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:11 1.00000000.inode [read 0~0] 1.232c0e14 e1710) v4 -- ?+0 0x1d0fd40 con 0x1d04b00
  189. -68> 2013-06-04 18:07:47.713229 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 3 ==== osd_op_reply(8 100.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 111+0+0 (1658436801 0 0) 0x1ca9400 con 0x1d04b00
  190. -67> 2013-06-04 18:07:47.713303 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 4 ==== osd_op_reply(9 100.00000000.inode [read 0~432] ondisk = 0) v4 ==== 117+0+432 (1629627978 0 1501311204) 0x1d22200 con 0x1d04b00
  191. -66> 2013-06-04 18:07:47.713502 7fadd0aa6700 5 mds.0.45 ms_handle_connect on 10.10.4.233:6800/8758
  192. -65> 2013-06-04 18:07:47.713523 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 5 ==== osd_op_reply(11 1.00000000.inode [read 0~398] ondisk = 0) v4 ==== 115+0+398 (3276024171 0 3300616070) 0x1ca9e00 con 0x1d04b00
  193. -64> 2013-06-04 18:07:47.714897 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 1 ==== osd_op_reply(10 1.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 109+0+0 (546655551 0 0) 0x1d22000 con 0x1d058c0
  194. -63> 2013-06-04 18:07:47.714964 7fadd0aa6700 2 mds.0.45 boot_start 3: replaying mds log
  195. -62> 2013-06-04 18:07:47.715169 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:12 200.00000072 [read 834~4193470] 1.6835c594 e1710) v4 -- ?+0 0x1d52fc0 con 0x1d04b00
  196. -61> 2013-06-04 18:07:47.715249 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:13 200.00000073 [read 0~4194304] 1.60f9d165 e1710) v4 -- ?+0 0x1d53440 con 0x1d04b00
  197. -60> 2013-06-04 18:07:47.715340 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:14 200.00000074 [read 0~4194304] 1.1a075cb8 e1710) v4 -- ?+0 0x1d53200 con 0x1cade40
  198. -59> 2013-06-04 18:07:47.715434 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6800/7185 -- osd_op(mds.0.45:15 200.00000075 [read 0~4194304] 1.ffd657f6 e1710) v4 -- ?+0 0x1d53b00 con 0x1d04000
  199. -58> 2013-06-04 18:07:47.715472 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:16 200.00000076 [read 0~4194304] 1.e1fad4a2 e1710) v4 -- ?+0 0x1d538c0 con 0x1d058c0
  200. -57> 2013-06-04 18:07:47.715522 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:17 200.00000077 [read 0~4194304] 1.731a8bca e1710) v4 -- ?+0 0x1d53680 con 0x1d058c0
  201. -56> 2013-06-04 18:07:47.715617 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6800/7185 -- osd_op(mds.0.45:18 200.00000078 [read 0~4194304] 1.c29d16b9 e1710) v4 -- ?+0 0x1d53b00 con 0x1d04000
  202. -55> 2013-06-04 18:07:47.715694 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:19 200.00000079 [read 0~4194304] 1.35fa4977 e1710) v4 -- ?+0 0x1d538c0 con 0x1d058c0
  203. -54> 2013-06-04 18:07:47.715738 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:20 200.0000007a [read 0~4194304] 1.f153a6a4 e1710) v4 -- ?+0 0x1d52b40 con 0x1d04840
  204. -53> 2013-06-04 18:07:47.715783 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:21 200.0000007b [read 0~4194304] 1.631a7623 e1710) v4 -- ?+0 0x1d5cfc0 con 0x1d04b00
  205. -52> 2013-06-04 18:07:47.715818 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:22 200.0000007c [read 0~4194304] 1.ec70f3f7 e1710) v4 -- ?+0 0x1d5cd80 con 0x1d04840
  206. -51> 2013-06-04 18:07:47.745808 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 6 ==== osd_op_reply(12 200.00000072 [read 834~4193470] ondisk = 0) v4 ==== 111+0+4193470 (3274224603 0 841000132) 0x1ca9c00 con 0x1d04b00
  207. -50> 2013-06-04 18:07:47.758680 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 7 ==== osd_op_reply(13 200.00000073 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (297063178 0 1049883112) 0x1d22e00 con 0x1d04b00
  208. -49> 2013-06-04 18:07:47.800591 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 8 ==== osd_op_reply(21 200.0000007b [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4275925845 0 1285928629) 0x1d22c00 con 0x1d04b00
  209. -48> 2013-06-04 18:07:47.800649 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6800/7185 -- osd_op(mds.0.45:23 200.0000007d [read 0~4194304] 1.e84f9969 e1710) v4 -- ?+0 0x1d5c900 con 0x1d04000
  210. -47> 2013-06-04 18:07:47.843958 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 3 ==== osd_op_reply(14 200.00000074 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1453785865 0 348870337) 0x1d22400 con 0x1cade40
  211. -46> 2013-06-04 18:07:47.846617 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.2 10.10.4.232:6800/7185 2 ==== osd_op_reply(15 200.00000075 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (448083966 0 2601733502) 0x1ca9e00 con 0x1d04000
  212. -45> 2013-06-04 18:07:47.853405 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:24 200.0000007e [read 0~4194304] 1.3df7fdc6 e1710) v4 -- ?+0 0x1d52480 con 0x1cade40
  213. -44> 2013-06-04 18:07:47.874034 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 2 ==== osd_op_reply(20 200.0000007a [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3122676580 0 753812023) 0x1d22200 con 0x1d04840
  214. -43> 2013-06-04 18:07:47.889705 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:25 200.0000007f [read 0~4194304] 1.8317f614 e1710) v4 -- ?+0 0x1d52000 con 0x1d04b00
  215. -42> 2013-06-04 18:07:47.907340 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 2 ==== osd_op_reply(17 200.00000077 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1039974638 0 1481779026) 0x1d22e00 con 0x1d058c0
  216. -41> 2013-06-04 18:07:47.907377 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 9 ==== osd_op_reply(25 200.0000007f [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2015639806 0 2053801285) 0x1d22a00 con 0x1d04b00
  217. -40> 2013-06-04 18:07:47.965777 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.2 10.10.4.232:6800/7185 3 ==== osd_op_reply(18 200.00000078 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3565772367 0 1686486247) 0x1ca9400 con 0x1d04000
  218. -39> 2013-06-04 18:07:47.996448 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 3 ==== osd_op_reply(22 200.0000007c [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1142952119 0 4123946153) 0x1ca9c00 con 0x1d04840
  219. -38> 2013-06-04 18:07:47.997445 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 4 ==== osd_op_reply(24 200.0000007e [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3100273842 0 3615660657) 0x1d22c00 con 0x1cade40
  220. -37> 2013-06-04 18:07:48.024367 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 3 ==== osd_op_reply(16 200.00000076 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4038986459 0 3423332489) 0x1d22000 con 0x1d058c0
  221. -36> 2013-06-04 18:07:48.024545 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.232:6800/7185 -- osd_op(mds.0.45:26 200.00000080 [read 0~4194304] 1.2f2f3339 e1710) v4 -- ?+0 0x1c99680 con 0x1d04000
  222. -35> 2013-06-04 18:07:48.045726 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.2 10.10.4.232:6800/7185 4 ==== osd_op_reply(23 200.0000007d [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (782659826 0 1868097600) 0x1d22e00 con 0x1d04000
  223. -34> 2013-06-04 18:07:48.082238 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:27 200.00000081 [read 0~4194304] 1.7614352c e1710) v4 -- ?+0 0x1d5dd40 con 0x1d04840
  224. -33> 2013-06-04 18:07:48.091608 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 4 ==== osd_op_reply(19 200.00000079 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3808101519 0 1803234917) 0x1d23200 con 0x1d058c0
  225. -32> 2013-06-04 18:07:48.108893 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.2 10.10.4.232:6800/7185 5 ==== osd_op_reply(26 200.00000080 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3737358763 0 4069336647) 0x1d22200 con 0x1d04000
  226. -31> 2013-06-04 18:07:48.120389 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:28 200.00000082 [read 0~4194304] 1.c892bcc0 e1710) v4 -- ?+0 0x1d5d8c0 con 0x1d04b00
  227. -30> 2013-06-04 18:07:48.163507 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 10 ==== osd_op_reply(28 200.00000082 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3933548445 0 714155040) 0x1d22800 con 0x1d04b00
  228. -29> 2013-06-04 18:07:48.163557 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 4 ==== osd_op_reply(27 200.00000081 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2043383298 0 3228061585) 0x1d23800 con 0x1d04840
  229. -28> 2013-06-04 18:07:48.165617 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.236:6800/2741 -- osd_op(mds.0.45:29 200.00000083 [read 0~4194304] 1.b41ca39d e1710) v4 -- ?+0 0x1d5d440 con 0x1d04420
  230. -27> 2013-06-04 18:07:48.220097 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.6 10.10.4.236:6800/2741 2 ==== osd_op_reply(29 200.00000083 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4211405560 0 2346240274) 0x1d22c00 con 0x1d04420
  231. -26> 2013-06-04 18:07:48.221863 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.236:6800/2741 -- osd_op(mds.0.45:30 200.00000084 [read 0~4194304] 1.e3a6e851 e1710) v4 -- ?+0 0x1d538c0 con 0x1d04420
  232. -25> 2013-06-04 18:07:48.260320 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:31 200.00000085 [read 0~4194304] 1.33afdd30 e1710) v4 -- ?+0 0x1d53440 con 0x1d058c0
  233. -24> 2013-06-04 18:07:48.270525 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.6 10.10.4.236:6800/2741 3 ==== osd_op_reply(30 200.00000084 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (896763281 0 3672634741) 0x1ca9c00 con 0x1d04420
  234. -23> 2013-06-04 18:07:48.298596 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:32 200.00000086 [read 0~4194304] 1.81c8766c e1710) v4 -- ?+0 0x1d52900 con 0x1d058c0
  235. -22> 2013-06-04 18:07:48.331435 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 5 ==== osd_op_reply(31 200.00000085 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4067861066 0 215900731) 0x1d23000 con 0x1d058c0
  236. -21> 2013-06-04 18:07:48.337551 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:33 200.00000087 [read 0~4194304] 1.7edfaacf e1710) v4 -- ?+0 0x1d53b00 con 0x1cade40
  237. -20> 2013-06-04 18:07:48.347754 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 6 ==== osd_op_reply(32 200.00000086 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (235135373 0 3187538869) 0x1d22a00 con 0x1d058c0
  238. -19> 2013-06-04 18:07:48.378343 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:34 200.00000088 [read 0~4194304] 1.2ada0bc0 e1710) v4 -- ?+0 0x1d5d8c0 con 0x1d04b00
  239. -18> 2013-06-04 18:07:48.386447 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 5 ==== osd_op_reply(33 200.00000087 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3554288293 0 4252798506) 0x1ca9400 con 0x1cade40
  240. -17> 2013-06-04 18:07:48.432800 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.236:6800/2741 -- osd_op(mds.0.45:35 200.00000089 [read 0~4194304] 1.bd81211b e1710) v4 -- ?+0 0x1d5d440 con 0x1d04420
  241. -16> 2013-06-04 18:07:48.459131 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 11 ==== osd_op_reply(34 200.00000088 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2211751308 0 2952766977) 0x1d22800 con 0x1d04b00
  242. -15> 2013-06-04 18:07:48.477233 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:36 200.0000008a [read 0~4194304] 1.89fedb1f e1710) v4 -- ?+0 0x1d53440 con 0x1d058c0
  243. -14> 2013-06-04 18:07:48.481119 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.6 10.10.4.236:6800/2741 4 ==== osd_op_reply(35 200.00000089 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (548950636 0 3313230351) 0x1d23c00 con 0x1d04420
  244. -13> 2013-06-04 18:07:48.523215 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.233:6800/8758 -- osd_op(mds.0.45:37 200.0000008b [read 0~4194304] 1.2c68723b e1710) v4 -- ?+0 0x1d0eb40 con 0x1d058c0
  245. -12> 2013-06-04 18:07:48.523578 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 7 ==== osd_op_reply(36 200.0000008a [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3778134084 0 1676190126) 0x1d23e00 con 0x1d058c0
  246. -11> 2013-06-04 18:07:48.570687 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.3 10.10.4.233:6800/8758 8 ==== osd_op_reply(37 200.0000008b [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (991416610 0 388315234) 0x383e800 con 0x1d058c0
  247. -10> 2013-06-04 18:07:48.582472 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.235:6800/4094 -- osd_op(mds.0.45:38 200.0000008c [read 0~4194304] 1.3b43fbac e1710) v4 -- ?+0 0x3719440 con 0x1cade40
  248. -9> 2013-06-04 18:07:48.620246 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:39 200.0000008d [read 0~4194304] 1.9bc3e37a e1710) v4 -- ?+0 0x3718fc0 con 0x1d04840
  249. -8> 2013-06-04 18:07:48.629064 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.5 10.10.4.235:6800/4094 6 ==== osd_op_reply(38 200.0000008c [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2474442685 0 1923815345) 0x1ca9e00 con 0x1cade40
  250. -7> 2013-06-04 18:07:48.668043 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.236:6800/2741 -- osd_op(mds.0.45:40 200.0000008e [read 0~4194304] 1.6e4748c3 e1710) v4 -- ?+0 0x3718b40 con 0x1d04420
  251. -6> 2013-06-04 18:07:48.669111 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 5 ==== osd_op_reply(39 200.0000008d [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3126619670 0 412669043) 0x1d23600 con 0x1d04840
  252. -5> 2013-06-04 18:07:48.716917 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.6 10.10.4.236:6800/2741 5 ==== osd_op_reply(40 200.0000008e [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (454973691 0 3494601257) 0x1d23a00 con 0x1d04420
  253. -4> 2013-06-04 18:07:48.721080 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.231:6801/8051 -- osd_op(mds.0.45:41 200.0000008f [read 0~4194304] 1.dd415afc e1710) v4 -- ?+0 0x37186c0 con 0x1d04b00
  254. -3> 2013-06-04 18:07:48.771786 7fadcd594700 1 -- 10.10.4.231:6800/11826 --> 10.10.4.234:6800/6714 -- osd_op(mds.0.45:42 200.00000090 [read 0~1557860] 1.13366537 e1710) v4 -- ?+0 0x3718240 con 0x1d04840
  255. -2> 2013-06-04 18:07:48.817891 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.1 10.10.4.231:6801/8051 12 ==== osd_op_reply(41 200.0000008f [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2977708972 0 3102128287) 0x1d22200 con 0x1d04b00
  256. -1> 2013-06-04 18:07:48.817942 7fadd0aa6700 1 -- 10.10.4.231:6800/11826 <== osd.4 10.10.4.234:6800/6714 6 ==== osd_op_reply(42 200.00000090 [read 0~1557860] ondisk = 0) v4 ==== 111+0+1557860 (697164660 0 821739504) 0x1d23400 con 0x1d04840
  257. 0> 2013-06-04 18:07:49.261299 7fadcd594700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7fadcd594700 time 2013-06-04 18:07:49.260303
  258. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  259.  
  260. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  261. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  262. 2: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  263. 3: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  264. 4: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  265. 5: (()+0x7f8e) [0x7fadd5c3ef8e]
  266. 6: (clone()+0x6d) [0x7fadd455be1d]
  267. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  268.  
  269. --- logging levels ---
  270. 0/ 5 none
  271. 0/ 1 lockdep
  272. 0/ 1 context
  273. 1/ 1 crush
  274. 1/ 5 mds
  275. 1/ 5 mds_balancer
  276. 1/ 5 mds_locker
  277. 1/ 5 mds_log
  278. 1/ 5 mds_log_expire
  279. 1/ 5 mds_migrator
  280. 0/ 1 buffer
  281. 0/ 1 timer
  282. 0/ 1 filer
  283. 0/ 1 striper
  284. 0/ 1 objecter
  285. 0/ 5 rados
  286. 0/ 5 rbd
  287. 0/ 5 journaler
  288. 0/ 5 objectcacher
  289. 0/ 5 client
  290. 0/ 5 osd
  291. 0/ 5 optracker
  292. 0/ 5 objclass
  293. 1/ 3 filestore
  294. 1/ 3 journal
  295. 0/ 5 ms
  296. 1/ 5 mon
  297. 0/10 monc
  298. 0/ 5 paxos
  299. 0/ 5 tp
  300. 1/ 5 auth
  301. 1/ 5 crypto
  302. 1/ 1 finisher
  303. 1/ 5 heartbeatmap
  304. 1/ 5 perfcounter
  305. 1/ 5 rgw
  306. 1/ 5 hadoop
  307. 1/ 5 javaclient
  308. 1/ 5 asok
  309. 1/ 1 throttle
  310. -2/-2 (syslog threshold)
  311. 99/99 (stderr threshold)
  312. max_recent 10000
  313. max_new 1000
  314. log_file
  315. --- end dump of recent events ---
  316. terminate called after throwing an instance of 'ceph::FailedAssertion'
  317. *** Caught signal (Aborted) **
  318. in thread 7fadcd594700
  319. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  320. 1: ceph-mds() [0x8662f0]
  321. 2: (()+0xfbd0) [0x7fadd5c46bd0]
  322. 3: (gsignal()+0x37) [0x7fadd4499037]
  323. 4: (abort()+0x148) [0x7fadd449c698]
  324. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fadd4da5e8d]
  325. 6: (()+0x5ef76) [0x7fadd4da3f76]
  326. 7: (()+0x5efa3) [0x7fadd4da3fa3]
  327. 8: (()+0x5f1de) [0x7fadd4da41de]
  328. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cbe3d]
  329. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  330. 11: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  331. 12: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  332. 13: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  333. 14: (()+0x7f8e) [0x7fadd5c3ef8e]
  334. 15: (clone()+0x6d) [0x7fadd455be1d]
  335. 2013-06-04 18:07:49.266784 7fadcd594700 -1 *** Caught signal (Aborted) **
  336. in thread 7fadcd594700
  337.  
  338. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  339. 1: ceph-mds() [0x8662f0]
  340. 2: (()+0xfbd0) [0x7fadd5c46bd0]
  341. 3: (gsignal()+0x37) [0x7fadd4499037]
  342. 4: (abort()+0x148) [0x7fadd449c698]
  343. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fadd4da5e8d]
  344. 6: (()+0x5ef76) [0x7fadd4da3f76]
  345. 7: (()+0x5efa3) [0x7fadd4da3fa3]
  346. 8: (()+0x5f1de) [0x7fadd4da41de]
  347. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cbe3d]
  348. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  349. 11: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  350. 12: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  351. 13: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  352. 14: (()+0x7f8e) [0x7fadd5c3ef8e]
  353. 15: (clone()+0x6d) [0x7fadd455be1d]
  354. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  355.  
  356. --- begin dump of recent events ---
  357. 0> 2013-06-04 18:07:49.266784 7fadcd594700 -1 *** Caught signal (Aborted) **
  358. in thread 7fadcd594700
  359.  
  360. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  361. 1: ceph-mds() [0x8662f0]
  362. 2: (()+0xfbd0) [0x7fadd5c46bd0]
  363. 3: (gsignal()+0x37) [0x7fadd4499037]
  364. 4: (abort()+0x148) [0x7fadd449c698]
  365. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fadd4da5e8d]
  366. 6: (()+0x5ef76) [0x7fadd4da3f76]
  367. 7: (()+0x5efa3) [0x7fadd4da3fa3]
  368. 8: (()+0x5f1de) [0x7fadd4da41de]
  369. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cbe3d]
  370. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3af5) [0x523235]
  371. 11: (EUpdate::replay(MDS*)+0x3a) [0x52bd9a]
  372. 12: (MDLog::_replay_thread()+0x5cf) [0x6f42cf]
  373. 13: (MDLog::ReplayThread::entry()+0xd) [0x505cbd]
  374. 14: (()+0x7f8e) [0x7fadd5c3ef8e]
  375. 15: (clone()+0x6d) [0x7fadd455be1d]
  376. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  377.  
  378. --- logging levels ---
  379. 0/ 5 none
  380. 0/ 1 lockdep
  381. 0/ 1 context
  382. 1/ 1 crush
  383. 1/ 5 mds
  384. 1/ 5 mds_balancer
  385. 1/ 5 mds_locker
  386. 1/ 5 mds_log
  387. 1/ 5 mds_log_expire
  388. 1/ 5 mds_migrator
  389. 0/ 1 buffer
  390. 0/ 1 timer
  391. 0/ 1 filer
  392. 0/ 1 striper
  393. 0/ 1 objecter
  394. 0/ 5 rados
  395. 0/ 5 rbd
  396. 0/ 5 journaler
  397. 0/ 5 objectcacher
  398. 0/ 5 client
  399. 0/ 5 osd
  400. 0/ 5 optracker
  401. 0/ 5 objclass
  402. 1/ 3 filestore
  403. 1/ 3 journal
  404. 0/ 5 ms
  405. 1/ 5 mon
  406. 0/10 monc
  407. 0/ 5 paxos
  408. 0/ 5 tp
  409. 1/ 5 auth
  410. 1/ 5 crypto
  411. 1/ 1 finisher
  412. 1/ 5 heartbeatmap
  413. 1/ 5 perfcounter
  414. 1/ 5 rgw
  415. 1/ 5 hadoop
  416. 1/ 5 javaclient
  417. 1/ 5 asok
  418. 1/ 1 throttle
  419. -2/-2 (syslog threshold)
  420. 99/99 (stderr threshold)
  421. max_recent 10000
  422. max_new 1000
  423. log_file
  424. --- end dump of recent events ---
  425. Aborted (core dumped)
  426. root@Ceph1:~#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!