API tools faq
paste
Advertisement
pveselov

ceph logs

pveselov
Mar 10th, 2014
265
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 64.52 KB | None | 0 0
raw download clone embed print report
  1. 2014-03-10 19:50:20.866511 7f4348e10700 0 mds.-1.0 ms_handle_connect on 10.16.20.11:6789/0
  2. 2014-03-10 19:50:20.933031 7f4348e10700 1 mds.-1.0 handle_mds_map standby
  3. 2014-03-10 19:50:21.007032 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  4. 2014-03-10 19:50:21.007062 7f4348e10700 1 mds.0.17 handle_mds_map state change up:standby --> up:replay
  5. 2014-03-10 19:50:21.007080 7f4348e10700 1 mds.0.17 replay_start
  6. 2014-03-10 19:50:21.007100 7f4348e10700 1 mds.0.17 recovery set is
  7. 2014-03-10 19:50:21.007104 7f4348e10700 1 mds.0.17 need osdmap epoch 8571, have 8570
  8. 2014-03-10 19:50:21.007108 7f4348e10700 1 mds.0.17 waiting for osdmap 8571 (which blacklists prior instance)
  9. 2014-03-10 19:50:21.007174 7f4348e10700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are
  10. 2014-03-10 19:50:21.030796 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.20.11:6801/10026
  11. 2014-03-10 19:50:21.033083 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.43.12:6801/20302
  12. 2014-03-10 19:50:21.362806 7f4348e10700 0 mds.0.cache creating system inode with ino:100
  13. 2014-03-10 19:50:21.363456 7f4348e10700 0 mds.0.cache creating system inode with ino:1
  14. 2014-03-10 19:50:22.090456 7f4348e10700 0 mds.0.17 ms_handle_reset on 10.16.43.12:6801/20302
  15. 2014-03-10 19:50:22.096882 7f4346207700 0 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
  16. 2014-03-10 19:50:40.895235 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.43.12:6802/15363
  17. 2014-03-10 19:50:42.620182 7f4346106700 1 mds.0.17 replay_done
  18. 2014-03-10 19:50:42.627286 7f4346106700 1 mds.0.17 making mds journal writeable
  19. 2014-03-10 19:50:42.666029 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  20. 2014-03-10 19:50:42.666036 7f4348e10700 1 mds.0.17 handle_mds_map state change up:replay --> up:reconnect
  21. 2014-03-10 19:50:42.666047 7f4348e10700 1 mds.0.17 reconnect_start
  22. 2014-03-10 19:50:42.666051 7f4348e10700 1 mds.0.17 reopen_log
  23. 2014-03-10 19:50:42.666076 7f4348e10700 1 mds.0.server reconnect_clients -- 10 sessions
  24. 2014-03-10 19:51:30.881953 7f4346d0b700 1 mds.0.server reconnect gave up on client.5312 10.16.20.14:0/1571
  25. 2014-03-10 19:51:30.881981 7f4346d0b700 1 mds.0.server reconnect gave up on client.5519 10.16.43.14:0/2173
  26. 2014-03-10 19:51:30.881987 7f4346d0b700 1 mds.0.server reconnect gave up on client.34698 10.16.20.11:0/9502
  27. 2014-03-10 19:51:30.881993 7f4346d0b700 1 mds.0.server reconnect gave up on client.40076 10.16.20.15:0/1567
  28. 2014-03-10 19:51:30.881997 7f4346d0b700 1 mds.0.server reconnect gave up on client.286652 10.16.20.12:0/3388
  29. 2014-03-10 19:51:30.882002 7f4346d0b700 1 mds.0.server reconnect gave up on client.286686 10.16.43.12:0/10209
  30. 2014-03-10 19:51:30.882007 7f4346d0b700 1 mds.0.server reconnect gave up on client.389212 10.16.43.16:0/31946
  31. 2014-03-10 19:51:30.882011 7f4346d0b700 1 mds.0.server reconnect gave up on client.394730 10.16.43.11:0/11285
  32. 2014-03-10 19:51:30.882016 7f4346d0b700 1 mds.0.server reconnect gave up on client.645558 10.16.20.13:0/2916
  33. 2014-03-10 19:51:30.882021 7f4346d0b700 1 mds.0.server reconnect gave up on client.647382 10.16.43.15:0/29413
  34. 2014-03-10 19:51:30.882027 7f4346d0b700 1 mds.0.17 reconnect_done
  35. 2014-03-10 19:51:30.928200 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  36. 2014-03-10 19:51:30.928206 7f4348e10700 1 mds.0.17 handle_mds_map state change up:reconnect --> up:rejoin
  37. 2014-03-10 19:51:30.928212 7f4348e10700 1 mds.0.17 rejoin_joint_start
  38. 2014-03-10 19:51:30.960503 7f4348e10700 -1 *** Caught signal (Segmentation fault) **
  39. in thread 7f4348e10700
  40.  
  41. ceph version 0.56.7 (14f23ab86b0058a8651895b3dc972a29459f3a33)
  42. 1: /usr/bin/ceph-mds() [0x813a91]
  43. 2: (()+0xf8e0) [0x7f434d4df8e0]
  44. 3: (SnapRealm::have_past_parents_open(snapid_t, snapid_t)+0x5a) [0x6be9da]
  45. 4: (MDCache::check_realm_past_parents(SnapRealm*)+0x2b) [0x55fe7b]
  46. 5: (MDCache::choose_lock_states_and_reconnect_caps()+0x29d) [0x567ddd]
  47. 6: (MDCache::rejoin_gather_finish()+0x91) [0x59da91]
  48. 7: (MDCache::rejoin_send_rejoins()+0x1b4f) [0x5a50bf]
  49. 8: (MDS::rejoin_joint_start()+0x13e) [0x4a718e]
  50. 9: (MDS::handle_mds_map(MMDSMap*)+0x2cda) [0x4bbf8a]
  51. 10: (MDS::handle_core_message(Message*)+0x93b) [0x4bdfeb]
  52. 11: (MDS::_dispatch(Message*)+0x2f) [0x4be0bf]
  53. 12: (MDS::ms_dispatch(Message*)+0x19b) [0x4bfc9b]
  54. 13: (DispatchQueue::entry()+0x309) [0x7e5cf9]
  55. 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7d607d]
  56. 15: (()+0x7c6b) [0x7f434d4d7c6b]
  57. 16: (clone()+0x6d) [0x7f434c6795ed]
  58. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  59.  
  60. --- begin dump of recent events ---
  61. -361> 2014-03-10 19:50:20.837247 7f434de6d780 5 asok(0x30269f0) register_command perfcounters_dump hook 0x3027be0
  62. -360> 2014-03-10 19:50:20.837330 7f434de6d780 5 asok(0x30269f0) register_command 1 hook 0x3027be0
  63. -359> 2014-03-10 19:50:20.837336 7f434de6d780 5 asok(0x30269f0) register_command perf dump hook 0x3027be0
  64. -358> 2014-03-10 19:50:20.837349 7f434de6d780 5 asok(0x30269f0) register_command perfcounters_schema hook 0x3027be0
  65. -357> 2014-03-10 19:50:20.837360 7f434de6d780 5 asok(0x30269f0) register_command 2 hook 0x3027be0
  66. -356> 2014-03-10 19:50:20.837364 7f434de6d780 5 asok(0x30269f0) register_command perf schema hook 0x3027be0
  67. -355> 2014-03-10 19:50:20.837368 7f434de6d780 5 asok(0x30269f0) register_command config show hook 0x3027be0
  68. -354> 2014-03-10 19:50:20.837373 7f434de6d780 5 asok(0x30269f0) register_command config set hook 0x3027be0
  69. -353> 2014-03-10 19:50:20.837378 7f434de6d780 5 asok(0x30269f0) register_command log flush hook 0x3027be0
  70. -352> 2014-03-10 19:50:20.837382 7f434de6d780 5 asok(0x30269f0) register_command log dump hook 0x3027be0
  71. -351> 2014-03-10 19:50:20.837386 7f434de6d780 5 asok(0x30269f0) register_command log reopen hook 0x3027be0
  72. -350> 2014-03-10 19:50:20.839558 7f434de6d780 0 ceph version 0.56.7 (14f23ab86b0058a8651895b3dc972a29459f3a33), process ceph-mds, pid 15080
  73. -349> 2014-03-10 19:50:20.839974 7f434de6d780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/15080 need_addr=1
  74. -348> 2014-03-10 19:50:20.847028 7f434de6d780 1 finished global_init_daemonize
  75. -347> 2014-03-10 19:50:20.856883 7f434de6d780 5 asok(0x30269f0) init /var/run/ceph/ceph-mds.d.asok
  76. -346> 2014-03-10 19:50:20.856947 7f434de6d780 5 asok(0x30269f0) bind_and_listen /var/run/ceph/ceph-mds.d.asok
  77. -345> 2014-03-10 19:50:20.857365 7f434de6d780 5 asok(0x30269f0) register_command 0 hook 0x3042df0
  78. -344> 2014-03-10 19:50:20.857402 7f434de6d780 5 asok(0x30269f0) register_command version hook 0x3042df0
  79. -343> 2014-03-10 19:50:20.857427 7f434de6d780 5 asok(0x30269f0) register_command git_version hook 0x3042df0
  80. -342> 2014-03-10 19:50:20.857433 7f434de6d780 5 asok(0x30269f0) register_command help hook 0x3043180
  81. -341> 2014-03-10 19:50:20.857550 7f434de6d780 10 monclient(hunting): build_initial_monmap
  82. -340> 2014-03-10 19:50:20.857703 7f434de6d780 1 -- 0.0.0.0:6800/15080 messenger.start
  83. -339> 2014-03-10 19:50:20.857955 7f434de6d780 5 adding auth protocol: cephx
  84. -338> 2014-03-10 19:50:20.857968 7f434de6d780 5 adding auth protocol: cephx
  85. -337> 2014-03-10 19:50:20.858199 7f434de6d780 1 accepter.accepter.start
  86. -336> 2014-03-10 19:50:20.858266 7f434de6d780 10 monclient(hunting): init
  87. -335> 2014-03-10 19:50:20.858275 7f434de6d780 5 adding auth protocol: cephx
  88. -334> 2014-03-10 19:50:20.858278 7f434de6d780 10 monclient(hunting): auth_supported 2 method cephx
  89. -333> 2014-03-10 19:50:20.858740 7f434de6d780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-d/keyring
  90. -332> 2014-03-10 19:50:20.858902 7f434de6d780 10 monclient(hunting): _reopen_session
  91. -331> 2014-03-10 19:50:20.859004 7f434de6d780 10 monclient(hunting): _pick_new_mon picked mon.c con 0x30496a0 addr 10.16.20.11:6789/0
  92. -330> 2014-03-10 19:50:20.859039 7f434de6d780 10 monclient(hunting): _send_mon_message to mon.c at 10.16.20.11:6789/0
  93. -329> 2014-03-10 19:50:20.859050 7f434de6d780 1 -- 0.0.0.0:6800/15080 --> 10.16.20.11:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x3049ae0 con 0x30496a0
  94. -328> 2014-03-10 19:50:20.859081 7f434de6d780 10 monclient(hunting): renew_subs
  95. -327> 2014-03-10 19:50:20.860970 7f434a613700 5 asok(0x30269f0) entry start
  96. -326> 2014-03-10 19:50:20.864084 7f4346e0c700 1 -- 10.16.43.12:6800/15080 learned my addr 10.16.43.12:6800/15080
  97. -325> 2014-03-10 19:50:20.866511 7f4348e10700 0 mds.-1.0 ms_handle_connect on 10.16.20.11:6789/0
  98. -324> 2014-03-10 19:50:20.870090 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 1 ==== mon_map v1 ==== 332+0+0 (1290947797 0 0) 0x7f4334000a80 con 0x30496a0
  99. -323> 2014-03-10 19:50:20.870130 7f4348e10700 10 monclient(hunting): handle_monmap mon_map v1
  100. -322> 2014-03-10 19:50:20.870148 7f4348e10700 10 monclient(hunting): got monmap 9, mon.c is now rank 0
  101. -321> 2014-03-10 19:50:20.870152 7f4348e10700 10 monclient(hunting): dump:
  102. epoch 9
  103. fsid 8df6d961-3531-46aa-a2dd-dbe41fc20084
  104. last_changed 2013-07-23 18:37:20.766151
  105. created 2013-04-06 09:09:37.063221
  106. 0: 10.16.20.11:6789/0 mon.c
  107. 1: 10.16.43.12:6789/0 mon.d
  108.  
  109. -320> 2014-03-10 19:50:20.870207 7f4348e10700 1 monclient(hunting): found mon.c
  110. -319> 2014-03-10 19:50:20.870347 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3449780625 0 0) 0x7f4334000d90 con 0x30496a0
  111. -318> 2014-03-10 19:50:20.870432 7f4348e10700 10 monclient: my global_id is 818366
  112. -317> 2014-03-10 19:50:20.870678 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  113. -316> 2014-03-10 19:50:20.870688 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f433c001540 con 0x30496a0
  114. -315> 2014-03-10 19:50:20.873598 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1620733898 0 0) 0x7f4334000a00 con 0x30496a0
  115. -314> 2014-03-10 19:50:20.873747 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  116. -313> 2014-03-10 19:50:20.873757 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f433c003650 con 0x30496a0
  117. -312> 2014-03-10 19:50:20.876695 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (3203448010 0 0) 0x7f4334000a00 con 0x30496a0
  118. -311> 2014-03-10 19:50:20.876819 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  119. -310> 2014-03-10 19:50:20.876827 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x3049dc0 con 0x30496a0
  120. -309> 2014-03-10 19:50:20.876892 7f4348e10700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-03-10 19:49:50.876886)
  121. -308> 2014-03-10 19:50:20.876905 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  122. -307> 2014-03-10 19:50:20.876910 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x7f433c002950 con 0x30496a0
  123. -306> 2014-03-10 19:50:20.876941 7f434de6d780 5 monclient: authenticate success, global_id 818366
  124. -305> 2014-03-10 19:50:20.876955 7f434de6d780 10 monclient: wait_auth_rotating waiting (until 2014-03-10 19:50:50.876951)
  125. -304> 2014-03-10 19:50:20.879627 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 5 ==== mon_map v1 ==== 332+0+0 (1290947797 0 0) 0x7f4334000fb0 con 0x30496a0
  126. -303> 2014-03-10 19:50:20.879661 7f4348e10700 10 monclient: handle_monmap mon_map v1
  127. -302> 2014-03-10 19:50:20.879673 7f4348e10700 10 monclient: got monmap 9, mon.c is now rank 0
  128. -301> 2014-03-10 19:50:20.879677 7f4348e10700 10 monclient: dump:
  129. epoch 9
  130. fsid 8df6d961-3531-46aa-a2dd-dbe41fc20084
  131. last_changed 2013-07-23 18:37:20.766151
  132. created 2013-04-06 09:09:37.063221
  133. 0: 10.16.20.11:6789/0 mon.c
  134. 1: 10.16.43.12:6789/0 mon.d
  135.  
  136. -300> 2014-03-10 19:50:20.879698 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f43340011c0 con 0x30496a0
  137. -299> 2014-03-10 19:50:20.879717 7f4348e10700 10 monclient: handle_subscribe_ack sent 2014-03-10 19:50:20.859084 renew after 2014-03-10 19:52:50.859084
  138. -298> 2014-03-10 19:50:20.879732 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (670714287 0 0) 0x7f43340014e0 con 0x30496a0
  139. -297> 2014-03-10 19:50:20.879801 7f4348e10700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:49:50.879799)
  140. -296> 2014-03-10 19:50:20.879833 7f434de6d780 10 monclient: wait_auth_rotating done
  141. -295> 2014-03-10 19:50:20.879866 7f434de6d780 5 asok(0x30269f0) register_command objecter_requests hook 0x304a4f0
  142. -294> 2014-03-10 19:50:20.879978 7f434de6d780 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  143. -293> 2014-03-10 19:50:20.879988 7f434de6d780 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:boot seq 1 v0) v2 -- ?+0 0x304ae90 con 0x30496a0
  144. -292> 2014-03-10 19:50:20.880034 7f434de6d780 10 monclient: renew_subs
  145. -291> 2014-03-10 19:50:20.880039 7f434de6d780 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  146. -290> 2014-03-10 19:50:20.880044 7f434de6d780 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({monmap=10+,osdmap=0}) v2 -- ?+0 0x304b820 con 0x30496a0
  147. -289> 2014-03-10 19:50:20.880055 7f434de6d780 10 monclient: renew_subs
  148. -288> 2014-03-10 19:50:20.880059 7f434de6d780 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  149. -287> 2014-03-10 19:50:20.880063 7f434de6d780 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({mdsmap=0+,monmap=10+,osdmap=0}) v2 -- ?+0 0x304bda0 con 0x30496a0
  150. -286> 2014-03-10 19:50:20.900650 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 8 ==== osd_map(8570..8570 src has 8070..8570) v3 ==== 3508+0+0 (1380696011 0 0) 0x7f4334000a30 con 0x30496a0
  151. -285> 2014-03-10 19:50:20.900902 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f43340010c0 con 0x30496a0
  152. -284> 2014-03-10 19:50:20.900918 7f4348e10700 10 monclient: handle_subscribe_ack sent 2014-03-10 19:50:20.880037 renew after 2014-03-10 19:52:50.880037
  153. -283> 2014-03-10 19:50:20.900929 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 10 ==== mdsmap(e 252) v1 ==== 535+0+0 (967346625 0 0) 0x7f43340024c0 con 0x30496a0
  154. -282> 2014-03-10 19:50:20.900952 7f4348e10700 5 mds.-1.0 handle_mds_map epoch 252 from mon.0
  155. -281> 2014-03-10 19:50:20.901015 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 11 ==== osd_map(8570..8570 src has 8070..8570) v3 ==== 3508+0+0 (1380696011 0 0) 0x7f4334003480 con 0x30496a0
  156. -280> 2014-03-10 19:50:20.901034 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f4334003730 con 0x30496a0
  157. -279> 2014-03-10 19:50:20.901045 7f4348e10700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  158. -278> 2014-03-10 19:50:20.932970 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 13 ==== mdsmap(e 253) v1 ==== 733+0+0 (1016082413 0 0) 0x7f43340017d0 con 0x30496a0
  159. -277> 2014-03-10 19:50:20.933004 7f4348e10700 5 mds.-1.-1 handle_mds_map epoch 253 from mon.0
  160. -276> 2014-03-10 19:50:20.933031 7f4348e10700 1 mds.-1.0 handle_mds_map standby
  161. -275> 2014-03-10 19:50:21.006947 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 14 ==== mdsmap(e 254) v1 ==== 535+0+0 (1941255870 0 0) 0x7f4334001700 con 0x30496a0
  162. -274> 2014-03-10 19:50:21.006991 7f4348e10700 5 mds.-1.0 handle_mds_map epoch 254 from mon.0
  163. -273> 2014-03-10 19:50:21.007023 7f4348e10700 1 -- 10.16.43.12:6800/15080 mark_down 10.16.43.12:6800/2934 -- pipe dne
  164. -272> 2014-03-10 19:50:21.007032 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  165. -271> 2014-03-10 19:50:21.007062 7f4348e10700 1 mds.0.17 handle_mds_map state change up:standby --> up:replay
  166. -270> 2014-03-10 19:50:21.007080 7f4348e10700 1 mds.0.17 replay_start
  167. -269> 2014-03-10 19:50:21.007100 7f4348e10700 1 mds.0.17 recovery set is
  168. -268> 2014-03-10 19:50:21.007104 7f4348e10700 1 mds.0.17 need osdmap epoch 8571, have 8570
  169. -267> 2014-03-10 19:50:21.007108 7f4348e10700 1 mds.0.17 waiting for osdmap 8571 (which blacklists prior instance)
  170. -266> 2014-03-10 19:50:21.007124 7f4348e10700 10 monclient: renew_subs
  171. -265> 2014-03-10 19:50:21.007132 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  172. -264> 2014-03-10 19:50:21.007139 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({mdsmap=255+,monmap=10+,osdmap=8571}) v2 -- ?+0 0x7f433c008420 con 0x30496a0
  173. -263> 2014-03-10 19:50:21.007174 7f4348e10700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are
  174. -262> 2014-03-10 19:50:21.007208 7f4348e10700 5 mds.0.migrator handle_mds_failure_or_stop mds.0
  175. -261> 2014-03-10 19:50:21.018912 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 15 ==== osd_map(8571..8571 src has 8070..8571) v3 ==== 308+0+0 (2800217208 0 0) 0x7f4334000a30 con 0x30496a0
  176. -260> 2014-03-10 19:50:21.019013 7f4348e10700 2 mds.0.17 boot_start 1: opening inotable
  177. -259> 2014-03-10 19:50:21.019276 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:1 mds0_inotable [read 0~0] 1.b852b893) v4 -- ?+0 0x7f433c009180 con 0x7f433c008e50
  178. -258> 2014-03-10 19:50:21.019323 7f4348e10700 2 mds.0.17 boot_start 1: opening sessionmap
  179. -257> 2014-03-10 19:50:21.019363 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:2 mds0_sessionmap [read 0~0] 1.3270c60b) v4 -- ?+0 0x7f433c009a00 con 0x7f433c008e50
  180. -256> 2014-03-10 19:50:21.019374 7f4348e10700 2 mds.0.17 boot_start 1: opening anchor table
  181. -255> 2014-03-10 19:50:21.019450 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:3 mds_anchortable [read 0~0] 1.a977f6a7) v4 -- ?+0 0x7f433c00a8b0 con 0x7f433c00a580
  182. -254> 2014-03-10 19:50:21.019465 7f4348e10700 2 mds.0.17 boot_start 1: opening snap table
  183. -253> 2014-03-10 19:50:21.019483 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:4 mds_snaptable [read 0~0] 1.d90270ad) v4 -- ?+0 0x7f433c00b1f0 con 0x7f433c008e50
  184. -252> 2014-03-10 19:50:21.019493 7f4348e10700 2 mds.0.17 boot_start 1: opening mds log
  185. -251> 2014-03-10 19:50:21.019505 7f4348e10700 5 mds.0.log open discovering log bounds
  186. -250> 2014-03-10 19:50:21.019515 7f4348e10700 1 mds.0.journaler(ro) recover start
  187. -249> 2014-03-10 19:50:21.019519 7f4348e10700 1 mds.0.journaler(ro) read_head
  188. -248> 2014-03-10 19:50:21.019545 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:5 200.00000000 [read 0~0] 1.844f3494) v4 -- ?+0 0x7f433c00c130 con 0x7f433c008e50
  189. -247> 2014-03-10 19:50:21.019569 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f4334003730 con 0x30496a0
  190. -246> 2014-03-10 19:50:21.019582 7f4348e10700 10 monclient: handle_subscribe_ack sent 2014-03-10 19:50:21.007128 renew after 2014-03-10 19:52:51.007128
  191. -245> 2014-03-10 19:50:21.030796 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.20.11:6801/10026
  192. -244> 2014-03-10 19:50:21.033083 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.43.12:6801/20302
  193. -243> 2014-03-10 19:50:21.065394 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 1 ==== osd_op_reply(1 mds0_inotable [read 0~29] = 0) v4 ==== 112+0+29 (3082723810 0 1526663537) 0x7f4320000a60 con 0x7f433c008e50
  194. -242> 2014-03-10 19:50:21.067361 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~21975] = 0) v4 ==== 114+0+21975 (356642019 0 2332820974) 0x7f4320002890 con 0x7f433c008e50
  195. -241> 2014-03-10 19:50:21.067826 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 3 ==== osd_op_reply(4 mds_snaptable [read 0~41] = 0) v4 ==== 112+0+41 (2993341024 0 2312406677) 0x7f4320002c30 con 0x7f433c008e50
  196. -240> 2014-03-10 19:50:21.079855 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 4 ==== osd_op_reply(5 200.00000000 [read 0~84] = 0) v4 ==== 111+0+84 (3622181752 0 3279650628) 0x7f4320002c30 con 0x7f433c008e50
  197. -239> 2014-03-10 19:50:21.079920 7f4348e10700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 9923723264, expire 9923724482, write 10052094370). probing for end of log (from 10052094370)...
  198. -238> 2014-03-10 19:50:21.079938 7f4348e10700 1 mds.0.journaler(ro) probing for end of the log
  199. -237> 2014-03-10 19:50:21.080043 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:6 200.0000095c [stat] 1.6b9c2ca rwordered) v4 -- ?+0 0x7f433c021110 con 0x7f433c00a580
  200. -236> 2014-03-10 19:50:21.080086 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:7 200.0000095d [stat] 1.b9045ba1 rwordered) v4 -- ?+0 0x7f433c0215b0 con 0x7f433c00a580
  201. -235> 2014-03-10 19:50:21.362618 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 1 ==== osd_op_reply(3 mds_anchortable [read 0~521] = 0) v4 ==== 114+0+521 (2596051907 0 3960091360) 0x7f4324000c50 con 0x7f433c00a580
  202. -234> 2014-03-10 19:50:21.362725 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 2 ==== osd_op_reply(6 200.0000095c [stat] = 0) v4 ==== 111+0+16 (327962699 0 145092094) 0x7f43240024a0 con 0x7f433c00a580
  203. -233> 2014-03-10 19:50:21.362757 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 3 ==== osd_op_reply(7 200.0000095d [stat] = -2 (No such file or directory)) v4 ==== 111+0+0 (2847608735 0 0) 0x7f43240027e0 con 0x7f433c00a580
  204. -232> 2014-03-10 19:50:21.362782 7f4348e10700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 10053302395 (header had 10052094370). recovered.
  205. -231> 2014-03-10 19:50:21.362791 7f4348e10700 2 mds.0.17 boot_start 2: loading/discovering base inodes
  206. -230> 2014-03-10 19:50:21.362806 7f4348e10700 0 mds.0.cache creating system inode with ino:100
  207. -229> 2014-03-10 19:50:21.363040 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:8 100.00000000 [getxattr inode] 1.c5265ab3) v4 -- ?+0 0x7f433c032940 con 0x7f433c008e50
  208. -228> 2014-03-10 19:50:21.363096 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:9 100.00000000.inode [read 0~0] 1.85dde07f) v4 -- ?+0 0x7f433c034190 con 0x7f433c008e50
  209. -227> 2014-03-10 19:50:21.363456 7f4348e10700 0 mds.0.cache creating system inode with ino:1
  210. -226> 2014-03-10 19:50:21.363535 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:10 1.00000000 [getxattr inode] 1.6b2cdaff) v4 -- ?+0 0x7f433c0349e0 con 0x7f433c008e50
  211. -225> 2014-03-10 19:50:21.363570 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:11 1.00000000.inode [read 0~0] 1.232c0e14) v4 -- ?+0 0x7f433c036270 con 0x7f433c00a580
  212. -224> 2014-03-10 19:50:21.364918 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 5 ==== osd_op_reply(8 100.00000000 [getxattr] = -61 (No data available)) v4 ==== 111+0+0 (409334422 0 0) 0x7f43200027e0 con 0x7f433c008e50
  213. -223> 2014-03-10 19:50:21.364966 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 6 ==== osd_op_reply(9 100.00000000.inode [read 0~398] = 0) v4 ==== 117+0+398 (299659235 0 3492920719) 0x7f4320002450 con 0x7f433c008e50
  214. -222> 2014-03-10 19:50:21.365064 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 7 ==== osd_op_reply(10 1.00000000 [getxattr] = -61 (No data available)) v4 ==== 109+0+0 (1512637288 0 0) 0x7f4320000a90 con 0x7f433c008e50
  215. -221> 2014-03-10 19:50:21.367261 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 4 ==== osd_op_reply(11 1.00000000.inode [read 0~398] = 0) v4 ==== 115+0+398 (1670017745 0 2087965404) 0x7f43240027e0 con 0x7f433c00a580
  216. -220> 2014-03-10 19:50:21.367307 7f4348e10700 2 mds.0.17 boot_start 3: replaying mds log
  217. -219> 2014-03-10 19:50:21.372338 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:12 200.0000093e [read 1218~4193086] 1.a7722bf4) v4 -- ?+0 0x7f4318000de0 con 0x7f433c008e50
  218. -218> 2014-03-10 19:50:21.372430 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:13 200.0000093f [read 0~4194304] 1.dc4d245f) v4 -- ?+0 0x7f4318001750 con 0x7f433c008e50
  219. -217> 2014-03-10 19:50:21.372971 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:14 200.00000940 [read 0~4194304] 1.619577d7) v4 -- ?+0 0x7f43180017b0 con 0x7f433c00a580
  220. -216> 2014-03-10 19:50:21.373147 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:15 200.00000941 [read 0~4194304] 1.d7fc3fc7) v4 -- ?+0 0x7f4318001ec0 con 0x7f433c00a580
  221. -215> 2014-03-10 19:50:21.373311 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:16 200.00000942 [read 0~4194304] 1.115803eb) v4 -- ?+0 0x7f4318002400 con 0x7f433c008e50
  222. -214> 2014-03-10 19:50:21.373582 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:17 200.00000943 [read 0~4194304] 1.c6905e84) v4 -- ?+0 0x7f4318002940 con 0x7f433c00a580
  223. -213> 2014-03-10 19:50:21.373741 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:18 200.00000944 [read 0~4194304] 1.65e11b90) v4 -- ?+0 0x7f4318002e40 con 0x7f433c008e50
  224. -212> 2014-03-10 19:50:21.374019 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:19 200.00000945 [read 0~4194304] 1.d1aad068) v4 -- ?+0 0x7f43180032a0 con 0x7f433c00a580
  225. -211> 2014-03-10 19:50:21.374164 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:20 200.00000946 [read 0~4194304] 1.e1dc427f) v4 -- ?+0 0x7f43180037d0 con 0x7f433c008e50
  226. -210> 2014-03-10 19:50:21.374433 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:21 200.00000947 [read 0~4194304] 1.2120022f) v4 -- ?+0 0x7f4318003d00 con 0x7f433c00a580
  227. -209> 2014-03-10 19:50:21.409153 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 8 ==== osd_op_reply(12 200.0000093e [read 1218~4193086] = 0) v4 ==== 111+0+4193086 (3246549297 0 1820962392) 0x7f43200022b0 con 0x7f433c008e50
  228. -208> 2014-03-10 19:50:21.459455 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 9 ==== osd_op_reply(13 200.0000093f [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1604199451 0 383326453) 0x7f4320000a30 con 0x7f433c008e50
  229. -207> 2014-03-10 19:50:21.492706 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 10 ==== osd_op_reply(18 200.00000944 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2657519758 0 757818183) 0x7f4320000a30 con 0x7f433c008e50
  230. -206> 2014-03-10 19:50:21.528321 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 11 ==== osd_op_reply(16 200.00000942 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2275016135 0 465926174) 0x7f4320000a30 con 0x7f433c008e50
  231. -205> 2014-03-10 19:50:21.561880 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6801/20302 12 ==== osd_op_reply(20 200.00000946 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1200835199 0 692318276) 0x7f4320000a30 con 0x7f433c008e50
  232. -204> 2014-03-10 19:50:21.586040 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 5 ==== osd_op_reply(14 200.00000940 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (543894354 0 3016902832) 0x7f43240024a0 con 0x7f433c00a580
  233. -203> 2014-03-10 19:50:21.642280 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:22 200.00000948 [read 0~4194304] 1.6d0097a1) v4 -- ?+0 0x7f431806d260 con 0x7f433c00a580
  234. -202> 2014-03-10 19:50:21.661485 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 6 ==== osd_op_reply(15 200.00000941 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (86557985 0 2998193638) 0x7f43240024a0 con 0x7f433c00a580
  235. -201> 2014-03-10 19:50:21.724604 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 7 ==== osd_op_reply(17 200.00000943 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3785290520 0 3880628549) 0x7f43240024a0 con 0x7f433c00a580
  236. -200> 2014-03-10 19:50:21.787574 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 8 ==== osd_op_reply(19 200.00000945 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3230512972 0 1887972756) 0x7f43240024a0 con 0x7f433c00a580
  237. -199> 2014-03-10 19:50:21.840255 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 9 ==== osd_op_reply(21 200.00000947 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (311367698 0 2444110192) 0x7f43240024a0 con 0x7f433c00a580
  238. -198> 2014-03-10 19:50:21.849640 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:23 200.00000949 [read 0~4194304] 1.d144bcc2) v4 -- ?+0 0x7f43181a7320 con 0x7f433c00a580
  239. -197> 2014-03-10 19:50:21.899238 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 10 ==== osd_op_reply(22 200.00000948 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (281725391 0 2572435248) 0x7f43240024a0 con 0x7f433c00a580
  240. -196> 2014-03-10 19:50:21.960109 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 11 ==== osd_op_reply(23 200.00000949 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2346119327 0 3315844379) 0x7f43240024a0 con 0x7f433c00a580
  241. -195> 2014-03-10 19:50:21.989616 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:24 200.0000094a [read 0~4194304] 1.850e1920) v4 -- ?+0 0x7f4318162c20 con 0x7f433c00a580
  242. -194> 2014-03-10 19:50:22.056112 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 12 ==== osd_op_reply(24 200.0000094a [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2483742438 0 3056391548) 0x7f43240024a0 con 0x7f433c00a580
  243. -193> 2014-03-10 19:50:22.069373 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:25 200.0000094b [read 0~4194304] 1.fbc96c5a) v4 -- ?+0 0x7f43181742b0 con 0x7f433c00a580
  244. -192> 2014-03-10 19:50:22.089956 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c008c00 sd=18 :56200 s=2 pgs=4 cs=1 l=1).reader couldn't read tag, Success
  245. -191> 2014-03-10 19:50:22.090013 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c008c00 sd=18 :56200 s=2 pgs=4 cs=1 l=1).fault 0: Success
  246. -190> 2014-03-10 19:50:22.090456 7f4348e10700 0 mds.0.17 ms_handle_reset on 10.16.43.12:6801/20302
  247. -189> 2014-03-10 19:50:22.090486 7f4348e10700 1 mds.0.objecter ms_handle_reset on osd.3
  248. -188> 2014-03-10 19:50:22.090492 7f4348e10700 1 -- 10.16.43.12:6800/15080 mark_down 0x7f433c008e50 -- pipe dne
  249. -187> 2014-03-10 19:50:22.090554 7f4348e10700 10 monclient: renew_subs
  250. -186> 2014-03-10 19:50:22.090563 7f4348e10700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  251. -185> 2014-03-10 19:50:22.090570 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({mdsmap=255+,monmap=10+,osdmap=8572}) v2 -- ?+0 0x7f433c033780 con 0x30496a0
  252. -184> 2014-03-10 19:50:22.096808 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  253. -183> 2014-03-10 19:50:22.096868 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  254. -182> 2014-03-10 19:50:22.096882 7f4346207700 0 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
  255. -181> 2014-03-10 19:50:22.096934 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  256. -180> 2014-03-10 19:50:22.096957 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  257. -179> 2014-03-10 19:50:22.120778 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:26 200.0000094c [read 0~4194304] 1.46819ec4) v4 -- ?+0 0x7f4318108920 con 0x7f433c00a580
  258. -178> 2014-03-10 19:50:22.149254 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 17 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f43340010c0 con 0x30496a0
  259. -177> 2014-03-10 19:50:22.149310 7f4348e10700 10 monclient: handle_subscribe_ack sent 2014-03-10 19:50:22.090559 renew after 2014-03-10 19:52:52.090559
  260. -176> 2014-03-10 19:50:22.153548 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 13 ==== osd_op_reply(25 200.0000094b [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1738085987 0 21179975) 0x7f43240024a0 con 0x7f433c00a580
  261. -175> 2014-03-10 19:50:22.201931 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 14 ==== osd_op_reply(26 200.0000094c [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3053992782 0 4258890485) 0x7f43240024a0 con 0x7f433c00a580
  262. -174> 2014-03-10 19:50:22.226004 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:27 200.0000094d [read 0~4194304] 1.edab1ee7) v4 -- ?+0 0x7f43181aaff0 con 0x7f433c033560
  263. -173> 2014-03-10 19:50:22.226183 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  264. -172> 2014-03-10 19:50:22.226217 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  265. -171> 2014-03-10 19:50:22.351014 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:28 200.0000094e [read 0~4194304] 1.5078c70c) v4 -- ?+0 0x7f43181fccf0 con 0x7f433c033560
  266. -170> 2014-03-10 19:50:22.359333 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  267. -169> 2014-03-10 19:50:22.359368 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  268. -168> 2014-03-10 19:50:22.446014 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:29 200.0000094f [read 0~4194304] 1.13ca80a3) v4 -- ?+0 0x7f4318608530 con 0x7f433c00a580
  269. -167> 2014-03-10 19:50:22.539467 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 15 ==== osd_op_reply(29 200.0000094f [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1687876642 0 1302845522) 0x7f43240024a0 con 0x7f433c00a580
  270. -166> 2014-03-10 19:50:22.556796 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:30 200.00000950 [read 0~4194304] 1.72a4c385) v4 -- ?+0 0x7f4318b76b00 con 0x7f433c00a580
  271. -165> 2014-03-10 19:50:22.630433 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 16 ==== osd_op_reply(30 200.00000950 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1262566027 0 3329332620) 0x7f43240024a0 con 0x7f433c00a580
  272. -164> 2014-03-10 19:50:22.658110 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6801/20302 -- osd_op(mds.0.17:31 200.00000951 [read 0~4194304] 1.a67c1240) v4 -- ?+0 0x7f4318a35250 con 0x7f433c033560
  273. -163> 2014-03-10 19:50:22.658284 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  274. -162> 2014-03-10 19:50:22.658314 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  275. -161> 2014-03-10 19:50:22.796061 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:32 200.00000952 [read 0~4194304] 1.bbcae1e3) v4 -- ?+0 0x7f4318cc9d30 con 0x7f433c00a580
  276. -160> 2014-03-10 19:50:22.871237 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 17 ==== osd_op_reply(32 200.00000952 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3398805509 0 680141612) 0x7f43240024a0 con 0x7f433c00a580
  277. -159> 2014-03-10 19:50:22.879031 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:33 200.00000953 [read 0~4194304] 1.efa09873) v4 -- ?+0 0x7f4318d58460 con 0x7f433c00a580
  278. -158> 2014-03-10 19:50:22.942856 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 18 ==== osd_op_reply(33 200.00000953 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1806462570 0 731537153) 0x7f43240024a0 con 0x7f433c00a580
  279. -157> 2014-03-10 19:50:22.954567 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:34 200.00000954 [read 0~4194304] 1.409252b) v4 -- ?+0 0x7f4318cca3f0 con 0x7f433c00a580
  280. -156> 2014-03-10 19:50:23.011832 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 19 ==== osd_op_reply(34 200.00000954 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1627296833 0 1616917334) 0x7f43240024a0 con 0x7f433c00a580
  281. -155> 2014-03-10 19:50:23.059957 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:35 200.00000955 [read 0~4194304] 1.48c1b78c) v4 -- ?+0 0x7f4318cb37c0 con 0x7f433c00a580
  282. -154> 2014-03-10 19:50:23.117335 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 20 ==== osd_op_reply(35 200.00000955 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2767474352 0 2637397483) 0x7f43240024a0 con 0x7f433c00a580
  283. -153> 2014-03-10 19:50:23.858963 7f4347e0e700 10 monclient: tick
  284. -152> 2014-03-10 19:50:23.859008 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:49:53.859004)
  285. -151> 2014-03-10 19:50:23.859028 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:50:23.859027; renew after: 2014-03-10 19:52:52.090559) -- no
  286. -150> 2014-03-10 19:50:24.258515 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.16.43.12:6801/20302, 111: Connection refused
  287. -149> 2014-03-10 19:50:24.258573 7f4346207700 2 -- 10.16.43.12:6800/15080 >> 10.16.43.12:6801/20302 pipe(0x7f433c033310 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
  288. -148> 2014-03-10 19:50:24.880186 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  289. -147> 2014-03-10 19:50:24.880231 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:replay seq 2 v254) v2 -- ?+0 0x7f4314000900 con 0x30496a0
  290. -146> 2014-03-10 19:50:24.883057 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 18 ==== mdsbeacon(818366/d up:replay seq 2 v254) v2 ==== 103+0+0 (4182496142 0 0) 0x7f43340014e0 con 0x30496a0
  291. -145> 2014-03-10 19:50:26.325999 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 19 ==== osd_map(8572..8572 src has 8071..8572) v3 ==== 169+0+0 (3731103223 0 0) 0x7f4334000a30 con 0x30496a0
  292. -144> 2014-03-10 19:50:26.326087 7f4348e10700 1 -- 10.16.43.12:6800/15080 mark_down 0x7f433c033560 -- 0x7f433c033310
  293. -143> 2014-03-10 19:50:26.326159 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:27 200.0000094d [read 0~4194304] 1.edab1ee7 RETRY) v4 -- ?+0 0x7f433c036f40 con 0x7f433c00a580
  294. -142> 2014-03-10 19:50:26.326193 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:28 200.0000094e [read 0~4194304] 1.5078c70c RETRY) v4 -- ?+0 0x7f433c037170 con 0x7f433c00a580
  295. -141> 2014-03-10 19:50:26.326347 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:31 200.00000951 [read 0~4194304] 1.a67c1240 RETRY) v4 -- ?+0 0x7f433c036f40 con 0x7f433c00a580
  296. -140> 2014-03-10 19:50:28.880380 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  297. -139> 2014-03-10 19:50:28.880407 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:replay seq 3 v254) v2 -- ?+0 0x7f4314000920 con 0x30496a0
  298. -138> 2014-03-10 19:50:28.883288 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 20 ==== mdsbeacon(818366/d up:replay seq 3 v254) v2 ==== 103+0+0 (2153950953 0 0) 0x7f43340014e0 con 0x30496a0
  299. -137> 2014-03-10 19:50:32.880549 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  300. -136> 2014-03-10 19:50:32.880578 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:replay seq 4 v254) v2 -- ?+0 0x7f4314000920 con 0x30496a0
  301. -135> 2014-03-10 19:50:32.883308 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 21 ==== mdsbeacon(818366/d up:replay seq 4 v254) v2 ==== 103+0+0 (3931154989 0 0) 0x7f43340014e0 con 0x30496a0
  302. -134> 2014-03-10 19:50:33.859174 7f4347e0e700 10 monclient: tick
  303. -133> 2014-03-10 19:50:33.859228 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:03.859225)
  304. -132> 2014-03-10 19:50:33.859251 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:50:33.859250; renew after: 2014-03-10 19:52:52.090559) -- no
  305. -131> 2014-03-10 19:50:36.880714 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  306. -130> 2014-03-10 19:50:36.880743 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:replay seq 5 v254) v2 -- ?+0 0x7f4314000920 con 0x30496a0
  307. -129> 2014-03-10 19:50:36.883529 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 22 ==== mdsbeacon(818366/d up:replay seq 5 v254) v2 ==== 103+0+0 (2474235722 0 0) 0x7f43340014e0 con 0x30496a0
  308. -128> 2014-03-10 19:50:40.880725 7f4346d0b700 10 monclient: renew_subs
  309. -127> 2014-03-10 19:50:40.880751 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  310. -126> 2014-03-10 19:50:40.880765 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mon_subscribe({mdsmap=255+,monmap=10+,osdmap=8573}) v2 -- ?+0 0x7f4314001990 con 0x30496a0
  311. -125> 2014-03-10 19:50:40.880794 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- ping v1 -- ?+0 0x7f4314001c20 con 0x7f433c00a580
  312. -124> 2014-03-10 19:50:40.881019 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  313. -123> 2014-03-10 19:50:40.881033 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:replay seq 6 v254) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  314. -122> 2014-03-10 19:50:40.883817 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 23 ==== osd_map(8573..8574 src has 8074..8574) v3 ==== 778+0+0 (4023834290 0 0) 0x7f4334000a30 con 0x30496a0
  315. -121> 2014-03-10 19:50:40.883995 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6802/15363 -- osd_op(mds.0.17:27 200.0000094d [read 0~4194304] 1.edab1ee7 RETRY) v4 -- ?+0 0x7f433c037900 con 0x7f433c0377b0
  316. -120> 2014-03-10 19:50:40.884056 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6802/15363 -- osd_op(mds.0.17:28 200.0000094e [read 0~4194304] 1.5078c70c RETRY) v4 -- ?+0 0x7f433c037dc0 con 0x7f433c0377b0
  317. -119> 2014-03-10 19:50:40.884081 7f4348e10700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6802/15363 -- osd_op(mds.0.17:31 200.00000951 [read 0~4194304] 1.a67c1240 RETRY) v4 -- ?+0 0x7f433c038280 con 0x7f433c0377b0
  318. -118> 2014-03-10 19:50:40.884116 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 24 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1112001777 0 0) 0x7f43340010c0 con 0x30496a0
  319. -117> 2014-03-10 19:50:40.884134 7f4348e10700 10 monclient: handle_subscribe_ack sent 2014-03-10 19:50:40.880746 renew after 2014-03-10 19:53:10.880746
  320. -116> 2014-03-10 19:50:40.884162 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 25 ==== mdsbeacon(818366/d up:replay seq 6 v254) v2 ==== 103+0+0 (402804963 0 0) 0x7f4334001800 con 0x30496a0
  321. -115> 2014-03-10 19:50:40.895235 7f4348e10700 0 mds.0.17 ms_handle_connect on 10.16.43.12:6802/15363
  322. -114> 2014-03-10 19:50:40.939912 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6802/15363 1 ==== osd_op_reply(27 200.0000094d [read 0~4194304] = 0) v4 ==== 111+0+4194304 (1458470235 0 4156464115) 0x7f430c000a30 con 0x7f433c0377b0
  323. -113> 2014-03-10 19:50:40.940060 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:36 200.00000956 [read 0~4194304] 1.5e7fd6ed) v4 -- ?+0 0x7f43181aaff0 con 0x7f433c00a580
  324. -112> 2014-03-10 19:50:40.969354 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6802/15363 2 ==== osd_op_reply(31 200.00000951 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3183621660 0 2957799033) 0x7f430c000a30 con 0x7f433c0377b0
  325. -111> 2014-03-10 19:50:40.995191 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6802/15363 3 ==== osd_op_reply(28 200.0000094e [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2416315032 0 1084400777) 0x7f430c000a30 con 0x7f433c0377b0
  326. -110> 2014-03-10 19:50:41.039906 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 21 ==== osd_op_reply(36 200.00000956 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (678257001 0 3927124870) 0x7f43240024a0 con 0x7f433c00a580
  327. -109> 2014-03-10 19:50:41.083547 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:37 200.00000957 [read 0~4194304] 1.620e38dc) v4 -- ?+0 0x7f4318dae1b0 con 0x7f433c00a580
  328. -108> 2014-03-10 19:50:41.154282 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 22 ==== osd_op_reply(37 200.00000957 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (478419011 0 3084353606) 0x7f43240024a0 con 0x7f433c00a580
  329. -107> 2014-03-10 19:50:41.198761 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:38 200.00000958 [read 0~4194304] 1.f98b6168) v4 -- ?+0 0x7f4318ec21e0 con 0x7f433c00a580
  330. -106> 2014-03-10 19:50:41.270816 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 23 ==== osd_op_reply(38 200.00000958 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (62068957 0 745653239) 0x7f43240024a0 con 0x7f433c00a580
  331. -105> 2014-03-10 19:50:41.311695 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6802/15363 -- osd_op(mds.0.17:39 200.00000959 [read 0~4194304] 1.71cdae8a) v4 -- ?+0 0x7f4319151020 con 0x7f433c0377b0
  332. -104> 2014-03-10 19:50:41.351269 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6802/15363 4 ==== osd_op_reply(39 200.00000959 [read 0~4194304] = 0) v4 ==== 111+0+4194304 (3479452830 0 2840427868) 0x7f430c0009d0 con 0x7f433c0377b0
  333. -103> 2014-03-10 19:50:41.452213 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:40 200.0000095a [read 0~4194304] 1.8eda8846) v4 -- ?+0 0x7f431915c030 con 0x7f433c00a580
  334. -102> 2014-03-10 19:50:41.523524 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 24 ==== osd_op_reply(40 200.0000095a [read 0~4194304] = 0) v4 ==== 111+0+4194304 (2242174365 0 569644435) 0x7f43240024a0 con 0x7f433c00a580
  335. -101> 2014-03-10 19:50:41.591576 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:41 200.0000095b [read 0~4194304] 1.aa60544d) v4 -- ?+0 0x7f43191a9bd0 con 0x7f433c00a580
  336. -100> 2014-03-10 19:50:41.655624 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 25 ==== osd_op_reply(41 200.0000095b [read 0~4194304] = 0) v4 ==== 111+0+4194304 (4168671489 0 522690164) 0x7f43240024a0 con 0x7f433c00a580
  337. -99> 2014-03-10 19:50:41.689182 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:42 200.0000095c [read 0~3750011] 1.6b9c2ca) v4 -- ?+0 0x7f43191d6e20 con 0x7f433c00a580
  338. -98> 2014-03-10 19:50:41.742190 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 26 ==== osd_op_reply(42 200.0000095c [read 0~3750011] = 0) v4 ==== 111+0+3750011 (2661977254 0 2070725605) 0x7f4324008920 con 0x7f433c00a580
  339. -97> 2014-03-10 19:50:42.620182 7f4346106700 1 mds.0.17 replay_done
  340. -96> 2014-03-10 19:50:42.627286 7f4346106700 1 mds.0.17 making mds journal writeable
  341. -95> 2014-03-10 19:50:42.627352 7f4346106700 1 mds.0.journaler(ro) set_writeable
  342. -94> 2014-03-10 19:50:42.627436 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:43 200.0000095c [zero 3750011~444293] 1.6b9c2ca) v4 -- ?+0 0x7f43195832a0 con 0x7f433c00a580
  343. -93> 2014-03-10 19:50:42.627518 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:44 200.0000095d [delete] 1.b9045ba1) v4 -- ?+0 0x7f4319583cc0 con 0x7f433c00a580
  344. -92> 2014-03-10 19:50:42.627568 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.43.12:6802/15363 -- osd_op(mds.0.17:45 200.0000095e [delete] 1.d1621b8f) v4 -- ?+0 0x7f43195a9fd0 con 0x7f433c0377b0
  345. -91> 2014-03-10 19:50:42.628750 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.3 10.16.43.12:6802/15363 5 ==== osd_op_reply(45 200.0000095e [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (3302836379 0 0) 0x7f430c0009d0 con 0x7f433c0377b0
  346. -90> 2014-03-10 19:50:42.628838 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:46 200.0000095f [delete] 1.bcd630ce) v4 -- ?+0 0x7f4319510e00 con 0x7f433c00a580
  347. -89> 2014-03-10 19:50:42.628991 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:47 200.00000960 [delete] 1.76981027) v4 -- ?+0 0x7f43195ae020 con 0x7f433c00a580
  348. -88> 2014-03-10 19:50:42.629145 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6801/10026 -- osd_op(mds.0.17:48 200.00000961 [delete] 1.83569ba3) v4 -- ?+0 0x7f431958ec70 con 0x7f433c00a580
  349. -87> 2014-03-10 19:50:42.629268 7f4346106700 2 mds.0.17 i am alone, moving to state reconnect
  350. -86> 2014-03-10 19:50:42.629303 7f4346106700 3 mds.0.17 request_state up:reconnect
  351. -85> 2014-03-10 19:50:42.629340 7f4346106700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  352. -84> 2014-03-10 19:50:42.629372 7f4346106700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 7 v254) v2 -- ?+0 0x7f43195ab800 con 0x30496a0
  353. -83> 2014-03-10 19:50:42.630953 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 27 ==== osd_op_reply(44 200.0000095d [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (1482037653 0 0) 0x7f43240008f0 con 0x7f433c00a580
  354. -82> 2014-03-10 19:50:42.631921 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 28 ==== osd_op_reply(46 200.0000095f [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (3078981183 0 0) 0x7f43240008f0 con 0x7f433c00a580
  355. -81> 2014-03-10 19:50:42.632249 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 29 ==== osd_op_reply(47 200.00000960 [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (1752431391 0 0) 0x7f43240008f0 con 0x7f433c00a580
  356. -80> 2014-03-10 19:50:42.632456 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 30 ==== osd_op_reply(48 200.00000961 [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (4273595756 0 0) 0x7f43240008f0 con 0x7f433c00a580
  357. -79> 2014-03-10 19:50:42.637182 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== osd.2 10.16.20.11:6801/10026 31 ==== osd_op_reply(43 200.0000095c [truncate 3750011] ondisk = 0) v4 ==== 111+0+0 (3218959238 0 0) 0x7f43240008f0 con 0x7f433c00a580
  358. -78> 2014-03-10 19:50:42.665937 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 26 ==== mdsmap(e 255) v1 ==== 535+0+0 (1357559449 0 0) 0x7f4334001700 con 0x30496a0
  359. -77> 2014-03-10 19:50:42.665967 7f4348e10700 5 mds.0.17 handle_mds_map epoch 255 from mon.0
  360. -76> 2014-03-10 19:50:42.666029 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  361. -75> 2014-03-10 19:50:42.666036 7f4348e10700 1 mds.0.17 handle_mds_map state change up:replay --> up:reconnect
  362. -74> 2014-03-10 19:50:42.666047 7f4348e10700 1 mds.0.17 reconnect_start
  363. -73> 2014-03-10 19:50:42.666051 7f4348e10700 1 mds.0.17 reopen_log
  364. -72> 2014-03-10 19:50:42.666076 7f4348e10700 1 mds.0.server reconnect_clients -- 10 sessions
  365. -71> 2014-03-10 19:50:42.666151 7f4348e10700 5 mds.0.bal rebalance done
  366. -70> 2014-03-10 19:50:43.859366 7f4347e0e700 10 monclient: tick
  367. -69> 2014-03-10 19:50:43.859412 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:13.859408)
  368. -68> 2014-03-10 19:50:43.859435 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:50:43.859434; renew after: 2014-03-10 19:53:10.880746) -- no
  369. -67> 2014-03-10 19:50:46.629574 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  370. -66> 2014-03-10 19:50:46.629604 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 8 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  371. -65> 2014-03-10 19:50:46.632321 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 27 ==== mdsbeacon(818366/d up:reconnect seq 8 v255) v2 ==== 103+0+0 (3838715492 0 0) 0x7f43340014e0 con 0x30496a0
  372. -64> 2014-03-10 19:50:50.629738 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  373. -63> 2014-03-10 19:50:50.629767 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 9 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  374. -62> 2014-03-10 19:50:50.632520 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 28 ==== mdsbeacon(818366/d up:reconnect seq 9 v255) v2 ==== 103+0+0 (2649185027 0 0) 0x7f43340014e0 con 0x30496a0
  375. -61> 2014-03-10 19:50:53.859558 7f4347e0e700 10 monclient: tick
  376. -60> 2014-03-10 19:50:53.859601 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:23.859597)
  377. -59> 2014-03-10 19:50:53.859623 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:50:53.859621; renew after: 2014-03-10 19:53:10.880746) -- no
  378. -58> 2014-03-10 19:50:54.629908 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  379. -57> 2014-03-10 19:50:54.629948 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 10 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  380. -56> 2014-03-10 19:50:54.632792 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 29 ==== mdsbeacon(818366/d up:reconnect seq 10 v255) v2 ==== 103+0+0 (379378858 0 0) 0x7f43340014e0 con 0x30496a0
  381. -55> 2014-03-10 19:50:58.630093 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  382. -54> 2014-03-10 19:50:58.630123 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 11 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  383. -53> 2014-03-10 19:50:58.632889 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 30 ==== mdsbeacon(818366/d up:reconnect seq 11 v255) v2 ==== 103+0+0 (1874174413 0 0) 0x7f43340014e0 con 0x30496a0
  384. -52> 2014-03-10 19:51:02.630264 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  385. -51> 2014-03-10 19:51:02.630303 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 12 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  386. -50> 2014-03-10 19:51:02.633220 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 31 ==== mdsbeacon(818366/d up:reconnect seq 12 v255) v2 ==== 103+0+0 (92775689 0 0) 0x7f43340014e0 con 0x30496a0
  387. -49> 2014-03-10 19:51:03.859746 7f4347e0e700 10 monclient: tick
  388. -48> 2014-03-10 19:51:03.859796 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:33.859792)
  389. -47> 2014-03-10 19:51:03.859821 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:51:03.859819; renew after: 2014-03-10 19:53:10.880746) -- no
  390. -46> 2014-03-10 19:51:06.630450 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  391. -45> 2014-03-10 19:51:06.630491 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 13 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  392. -44> 2014-03-10 19:51:06.633386 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 32 ==== mdsbeacon(818366/d up:reconnect seq 13 v255) v2 ==== 103+0+0 (2091833454 0 0) 0x7f43340014e0 con 0x30496a0
  393. -43> 2014-03-10 19:51:10.630636 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  394. -42> 2014-03-10 19:51:10.630675 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 14 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  395. -41> 2014-03-10 19:51:10.633521 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 33 ==== mdsbeacon(818366/d up:reconnect seq 14 v255) v2 ==== 103+0+0 (4157956039 0 0) 0x7f43340014e0 con 0x30496a0
  396. -40> 2014-03-10 19:51:13.859944 7f4347e0e700 10 monclient: tick
  397. -39> 2014-03-10 19:51:13.859994 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:43.859990)
  398. -38> 2014-03-10 19:51:13.860018 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:51:13.860017; renew after: 2014-03-10 19:53:10.880746) -- no
  399. -37> 2014-03-10 19:51:14.630821 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  400. -36> 2014-03-10 19:51:14.630861 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 15 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  401. -35> 2014-03-10 19:51:14.633737 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 34 ==== mdsbeacon(818366/d up:reconnect seq 15 v255) v2 ==== 103+0+0 (2398888608 0 0) 0x7f43340014e0 con 0x30496a0
  402. -34> 2014-03-10 19:51:18.631008 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  403. -33> 2014-03-10 19:51:18.631046 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 16 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  404. -32> 2014-03-10 19:51:18.633853 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 35 ==== mdsbeacon(818366/d up:reconnect seq 16 v255) v2 ==== 103+0+0 (2829304040 0 0) 0x7f43340014e0 con 0x30496a0
  405. -31> 2014-03-10 19:51:22.631182 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  406. -30> 2014-03-10 19:51:22.631221 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 17 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  407. -29> 2014-03-10 19:51:22.633963 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 36 ==== mdsbeacon(818366/d up:reconnect seq 17 v255) v2 ==== 103+0+0 (3515530639 0 0) 0x7f43340014e0 con 0x30496a0
  408. -28> 2014-03-10 19:51:23.860132 7f4347e0e700 10 monclient: tick
  409. -27> 2014-03-10 19:51:23.860183 7f4347e0e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-03-10 19:50:53.860179)
  410. -26> 2014-03-10 19:51:23.860208 7f4347e0e700 10 monclient: renew subs? (now: 2014-03-10 19:51:23.860206; renew after: 2014-03-10 19:53:10.880746) -- no
  411. -25> 2014-03-10 19:51:26.631373 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  412. -24> 2014-03-10 19:51:26.631403 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 18 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  413. -23> 2014-03-10 19:51:26.634088 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 37 ==== mdsbeacon(818366/d up:reconnect seq 18 v255) v2 ==== 103+0+0 (1525758502 0 0) 0x7f43340014e0 con 0x30496a0
  414. -22> 2014-03-10 19:51:30.631536 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  415. -21> 2014-03-10 19:51:30.631576 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:reconnect seq 19 v255) v2 -- ?+0 0x7f4314001f00 con 0x30496a0
  416. -20> 2014-03-10 19:51:30.634397 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 38 ==== mdsbeacon(818366/d up:reconnect seq 19 v255) v2 ==== 103+0+0 (601376577 0 0) 0x7f43340014e0 con 0x30496a0
  417. -19> 2014-03-10 19:51:30.881953 7f4346d0b700 1 mds.0.server reconnect gave up on client.5312 10.16.20.14:0/1571
  418. -18> 2014-03-10 19:51:30.881981 7f4346d0b700 1 mds.0.server reconnect gave up on client.5519 10.16.43.14:0/2173
  419. -17> 2014-03-10 19:51:30.881987 7f4346d0b700 1 mds.0.server reconnect gave up on client.34698 10.16.20.11:0/9502
  420. -16> 2014-03-10 19:51:30.881993 7f4346d0b700 1 mds.0.server reconnect gave up on client.40076 10.16.20.15:0/1567
  421. -15> 2014-03-10 19:51:30.881997 7f4346d0b700 1 mds.0.server reconnect gave up on client.286652 10.16.20.12:0/3388
  422. -14> 2014-03-10 19:51:30.882002 7f4346d0b700 1 mds.0.server reconnect gave up on client.286686 10.16.43.12:0/10209
  423. -13> 2014-03-10 19:51:30.882007 7f4346d0b700 1 mds.0.server reconnect gave up on client.389212 10.16.43.16:0/31946
  424. -12> 2014-03-10 19:51:30.882011 7f4346d0b700 1 mds.0.server reconnect gave up on client.394730 10.16.43.11:0/11285
  425. -11> 2014-03-10 19:51:30.882016 7f4346d0b700 1 mds.0.server reconnect gave up on client.645558 10.16.20.13:0/2916
  426. -10> 2014-03-10 19:51:30.882021 7f4346d0b700 1 mds.0.server reconnect gave up on client.647382 10.16.43.15:0/29413
  427. -9> 2014-03-10 19:51:30.882027 7f4346d0b700 1 mds.0.17 reconnect_done
  428. -8> 2014-03-10 19:51:30.882030 7f4346d0b700 3 mds.0.17 request_state up:rejoin
  429. -7> 2014-03-10 19:51:30.882045 7f4346d0b700 10 monclient: _send_mon_message to mon.c at 10.16.20.11:6789/0
  430. -6> 2014-03-10 19:51:30.882067 7f4346d0b700 1 -- 10.16.43.12:6800/15080 --> 10.16.20.11:6789/0 -- mdsbeacon(818366/d up:rejoin seq 20 v255) v2 -- ?+0 0x7f4314004280 con 0x30496a0
  431. -5> 2014-03-10 19:51:30.928117 7f4348e10700 1 -- 10.16.43.12:6800/15080 <== mon.0 10.16.20.11:6789/0 39 ==== mdsmap(e 256) v1 ==== 535+0+0 (1015733359 0 0) 0x7f4334001700 con 0x30496a0
  432. -4> 2014-03-10 19:51:30.928158 7f4348e10700 5 mds.0.17 handle_mds_map epoch 256 from mon.0
  433. -3> 2014-03-10 19:51:30.928200 7f4348e10700 1 mds.0.17 handle_mds_map i am now mds.0.17
  434. -2> 2014-03-10 19:51:30.928206 7f4348e10700 1 mds.0.17 handle_mds_map state change up:reconnect --> up:rejoin
  435. -1> 2014-03-10 19:51:30.928212 7f4348e10700 1 mds.0.17 rejoin_joint_start
  436. 0> 2014-03-10 19:51:30.960503 7f4348e10700 -1 *** Caught signal (Segmentation fault) **
  437. in thread 7f4348e10700
  438.  
  439. ceph version 0.56.7 (14f23ab86b0058a8651895b3dc972a29459f3a33)
  440. 1: /usr/bin/ceph-mds() [0x813a91]
  441. 2: (()+0xf8e0) [0x7f434d4df8e0]
  442. 3: (SnapRealm::have_past_parents_open(snapid_t, snapid_t)+0x5a) [0x6be9da]
  443. 4: (MDCache::check_realm_past_parents(SnapRealm*)+0x2b) [0x55fe7b]
  444. 5: (MDCache::choose_lock_states_and_reconnect_caps()+0x29d) [0x567ddd]
  445. 6: (MDCache::rejoin_gather_finish()+0x91) [0x59da91]
  446. 7: (MDCache::rejoin_send_rejoins()+0x1b4f) [0x5a50bf]
  447. 8: (MDS::rejoin_joint_start()+0x13e) [0x4a718e]
  448. 9: (MDS::handle_mds_map(MMDSMap*)+0x2cda) [0x4bbf8a]
  449. 10: (MDS::handle_core_message(Message*)+0x93b) [0x4bdfeb]
  450. 11: (MDS::_dispatch(Message*)+0x2f) [0x4be0bf]
  451. 12: (MDS::ms_dispatch(Message*)+0x19b) [0x4bfc9b]
  452. 13: (DispatchQueue::entry()+0x309) [0x7e5cf9]
  453. 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7d607d]
  454. 15: (()+0x7c6b) [0x7f434d4d7c6b]
  455. 16: (clone()+0x6d) [0x7f434c6795ed]
  456. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  457.  
  458. --- logging levels ---
  459. 0/ 5 none
  460. 0/ 1 lockdep
  461. 0/ 1 context
  462. 1/ 1 crush
  463. 1/ 5 mds
  464. 1/ 5 mds_balancer
  465. 1/ 5 mds_locker
  466. 1/ 5 mds_log
  467. 1/ 5 mds_log_expire
  468. 1/ 5 mds_migrator
  469. 0/ 1 buffer
  470. 0/ 1 timer
  471. 0/ 1 filer
  472. 0/ 1 striper
  473. 0/ 1 objecter
  474. 0/ 5 rados
  475. 0/ 5 rbd
  476. 0/ 5 journaler
  477. 0/ 5 objectcacher
  478. 0/ 5 client
  479. 0/ 5 osd
  480. 0/ 5 optracker
  481. 0/ 5 objclass
  482. 1/ 3 filestore
  483. 1/ 3 journal
  484. 0/ 5 ms
  485. 1/ 5 mon
  486. 0/10 monc
  487. 0/ 5 paxos
  488. 0/ 5 tp
  489. 1/ 5 auth
  490. 1/ 5 crypto
  491. 1/ 1 finisher
  492. 1/ 5 heartbeatmap
  493. 1/ 5 perfcounter
  494. 1/ 5 rgw
  495. 1/ 5 hadoop
  496. 1/ 5 javaclient
  497. 1/ 5 asok
  498. 1/ 1 throttle
  499. -2/-2 (syslog threshold)
  500. -1/-1 (stderr threshold)
  501. max_recent 10000
  502. max_new 1000
  503. log_file /var/log/ceph/ceph-mds.d.log
  504. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!