API tools faq
paste
Advertisement
bayphotomike

ceph-mon-crash-after-mapset

bayphotomike
Jan 14th, 2016
124
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 178.33 KB | None | 0 0
raw download clone embed print report
  1. 2016-01-14 15:50:21.295158 7f960d8f9700 0 mon.lts-osd2@1(leader) e2 handle_command mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1
  2. 2016-01-14 15:50:21.295198 7f960d8f9700 0 log_channel(audit) log [INF] : from='client.614112 :/0' entity='unknown.' cmd=[{"prefix": "mds setmap", "epoch": 7821}]: dispatch
  3. 2016-01-14 15:50:21.362013 7f960d8f9700 0 log_channel(cluster) log [INF] : monmap e2: 3 mons at {lts-mon=10.5.68.236:6789/0,lts-osd1=10.5.68.229:6789/0,lts-osd2=10.5.68.230:6789/0}
  4. 2016-01-14 15:50:21.362099 7f960d8f9700 0 log_channel(cluster) log [INF] : pgmap v6715571: 4192 pgs: 4 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 4185 active+clean; 31748 GB data, 96182 GB used, 273 TB / 367 TB avail
  5. 2016-01-14 15:50:21.362159 7f960d8f9700 0 log_channel(cluster) log [INF] : mdsmap e7932: 1/2/1 up {0=lts-osd1=up:resolve}, 1 up:standby
  6. 2016-01-14 15:50:21.362270 7f960d8f9700 0 log_channel(cluster) log [INF] : osdmap e10235: 102 osds: 101 up, 101 in
  7. 2016-01-14 15:50:21.365929 7f960d8f9700 -1 *** Caught signal (Aborted) **
  8. in thread 7f960d8f9700
  9.  
  10. ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
  11. 1: (()+0x68e43a) [0x7f961728643a]
  12. 2: (()+0x10340) [0x7f9615cde340]
  13. 3: (gsignal()+0x39) [0x7f961417fcc9]
  14. 4: (abort()+0x148) [0x7f96141830d8]
  15. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9614a8a535]
  16. 6: (()+0x5e6d6) [0x7f9614a886d6]
  17. 7: (()+0x5e703) [0x7f9614a88703]
  18. 8: (()+0x5e922) [0x7f9614a88922]
  19. 9: (ceph::buffer::list::iterator_impl<false>::copy(unsigned int, char*)+0xa5) [0x7f9617062675]
  20. 10: (MDSMap::decode(ceph::buffer::list::iterator&)+0x29) [0x7f9616fbb239]
  21. 11: (MDSMonitor::filesystem_command(std::shared_ptr<MonOpRequest>, std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x6dc) [0x7f9616eef14c]
  22. 12: (MDSMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x8e3) [0x7f9616ef1a73]
  23. 13: (MDSMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x2e4) [0x7f9616ef2334]
  24. 14: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xd7b) [0x7f9616e729db]
  25. 15: (PaxosService::C_RetryMessage::_finish(int)+0x54) [0x7f9616e75054]
  26. 16: (C_MonOp::finish(int)+0x69) [0x7f9616e42199]
  27. 17: (Context::complete(int)+0x9) [0x7f9616e412a9]
  28. 18: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7f9616e47364]
  29. 19: (Paxos::finish_round()+0x10b) [0x7f9616e671eb]
  30. 20: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xea2) [0x7f9616e6b632]
  31. 21: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x2d4) [0x7f9616e6bf24]
  32. 22: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xcfb) [0x7f9616e3c48b]
  33. 23: (Monitor::_ms_dispatch(Message*)+0x628) [0x7f9616e3cd78]
  34. 24: (Monitor::ms_dispatch(Message*)+0x23) [0x7f9616e5b6a3]
  35. 25: (DispatchQueue::entry()+0x63a) [0x7f96171d72aa]
  36. 26: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f961703f06d]
  37. 27: (()+0x8182) [0x7f9615cd6182]
  38. 28: (clone()+0x6d) [0x7f961424347d]
  39. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  40.  
  41. --- begin dump of recent events ---
  42. -954> 2016-01-14 15:50:16.171059 7f96175cf8c0 5 asok(0x7f961acf8000) register_command perfcounters_dump hook 0x7f961ac5c050
  43. -953> 2016-01-14 15:50:16.171088 7f96175cf8c0 5 asok(0x7f961acf8000) register_command 1 hook 0x7f961ac5c050
  44. -952> 2016-01-14 15:50:16.171092 7f96175cf8c0 5 asok(0x7f961acf8000) register_command perf dump hook 0x7f961ac5c050
  45. -951> 2016-01-14 15:50:16.171094 7f96175cf8c0 5 asok(0x7f961acf8000) register_command perfcounters_schema hook 0x7f961ac5c050
  46. -950> 2016-01-14 15:50:16.171101 7f96175cf8c0 5 asok(0x7f961acf8000) register_command 2 hook 0x7f961ac5c050
  47. -949> 2016-01-14 15:50:16.171103 7f96175cf8c0 5 asok(0x7f961acf8000) register_command perf schema hook 0x7f961ac5c050
  48. -948> 2016-01-14 15:50:16.171108 7f96175cf8c0 5 asok(0x7f961acf8000) register_command perf reset hook 0x7f961ac5c050
  49. -947> 2016-01-14 15:50:16.171110 7f96175cf8c0 5 asok(0x7f961acf8000) register_command config show hook 0x7f961ac5c050
  50. -946> 2016-01-14 15:50:16.171112 7f96175cf8c0 5 asok(0x7f961acf8000) register_command config set hook 0x7f961ac5c050
  51. -945> 2016-01-14 15:50:16.171114 7f96175cf8c0 5 asok(0x7f961acf8000) register_command config get hook 0x7f961ac5c050
  52. -944> 2016-01-14 15:50:16.171118 7f96175cf8c0 5 asok(0x7f961acf8000) register_command config diff hook 0x7f961ac5c050
  53. -943> 2016-01-14 15:50:16.171122 7f96175cf8c0 5 asok(0x7f961acf8000) register_command log flush hook 0x7f961ac5c050
  54. -942> 2016-01-14 15:50:16.171124 7f96175cf8c0 5 asok(0x7f961acf8000) register_command log dump hook 0x7f961ac5c050
  55. -941> 2016-01-14 15:50:16.171128 7f96175cf8c0 5 asok(0x7f961acf8000) register_command log reopen hook 0x7f961ac5c050
  56. -940> 2016-01-14 15:50:16.175135 7f96175cf8c0 0 set uid:gid to 64045:64045
  57. -939> 2016-01-14 15:50:16.175216 7f96175cf8c0 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-mon, pid 14401
  58. -938> 2016-01-14 15:50:16.176892 7f96175cf8c0 5 asok(0x7f961acf8000) init /var/run/ceph/ceph-mon.lts-osd2.asok
  59. -937> 2016-01-14 15:50:16.176906 7f96175cf8c0 5 asok(0x7f961acf8000) bind_and_listen /var/run/ceph/ceph-mon.lts-osd2.asok
  60. -936> 2016-01-14 15:50:16.176930 7f96175cf8c0 5 asok(0x7f961acf8000) register_command 0 hook 0x7f961ac580b8
  61. -935> 2016-01-14 15:50:16.176936 7f96175cf8c0 5 asok(0x7f961acf8000) register_command version hook 0x7f961ac580b8
  62. -934> 2016-01-14 15:50:16.176938 7f96175cf8c0 5 asok(0x7f961acf8000) register_command git_version hook 0x7f961ac580b8
  63. -933> 2016-01-14 15:50:16.176943 7f96175cf8c0 5 asok(0x7f961acf8000) register_command help hook 0x7f961ac5c240
  64. -932> 2016-01-14 15:50:16.176946 7f96175cf8c0 5 asok(0x7f961acf8000) register_command get_command_descriptions hook 0x7f961ac5c230
  65. -931> 2016-01-14 15:50:16.176976 7f9611720700 5 asok(0x7f961acf8000) entry start
  66. -930> 2016-01-14 15:50:16.189912 7f96175cf8c0 0 starting mon.lts-osd2 rank 1 at 10.5.68.230:6789/0 mon_data /var/lib/ceph/mon/ceph-lts-osd2 fsid cabd1728-2eca-4e18-a581-b4885364e5a4
  67. -929> 2016-01-14 15:50:16.189971 7f96175cf8c0 1 -- 10.5.68.230:6789/0 learned my addr 10.5.68.230:6789/0
  68. -928> 2016-01-14 15:50:16.189977 7f96175cf8c0 1 accepter.accepter.bind my_inst.addr is 10.5.68.230:6789/0 need_addr=0
  69. -927> 2016-01-14 15:50:16.190020 7f96175cf8c0 5 adding auth protocol: cephx
  70. -926> 2016-01-14 15:50:16.190026 7f96175cf8c0 5 adding auth protocol: cephx
  71. -925> 2016-01-14 15:50:16.190048 7f96175cf8c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  72. -924> 2016-01-14 15:50:16.190053 7f96175cf8c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info)
  73. -923> 2016-01-14 15:50:16.190127 7f96175cf8c0 1 mon.lts-osd2@-1(probing) e2 preinit fsid cabd1728-2eca-4e18-a581-b4885364e5a4
  74. -922> 2016-01-14 15:50:16.190512 7f96175cf8c0 1 mon.lts-osd2@-1(probing).paxosservice(pgmap 6714857..6715571) refresh upgraded, format 0 -> 1
  75. -921> 2016-01-14 15:50:16.190526 7f96175cf8c0 1 mon.lts-osd2@-1(probing).pg v0 on_upgrade discarding in-core PGMap
  76. -920> 2016-01-14 15:50:16.219532 7f96175cf8c0 4 mon.lts-osd2@-1(probing).mds e7932 new map
  77. -919> 2016-01-14 15:50:16.219546 7f96175cf8c0 0 mon.lts-osd2@-1(probing).mds e7932 print_map
  78. epoch 7932
  79. flags 0
  80. created 2015-06-16 12:48:55.442888
  81. modified 2016-01-14 15:43:22.740816
  82. tableserver 0
  83. root 0
  84. session_timeout 60
  85. session_autoclose 300
  86. max_file_size 1099511627776
  87. last_failure 7099
  88. last_failure_osd_epoch 10232
  89. compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
  90. max_mds 1
  91. in 0,1
  92. up {0=504996}
  93. failed
  94. damaged
  95. stopped
  96. data_pools 1
  97. metadata_pool 2
  98. inline_data disabled
  99. 505032: 10.5.68.236:6800/23788 'lts-mon' mds.-1.0 up:standby seq 2339
  100. 504996: 10.5.68.229:6800/14449 'lts-osd1' mds.0.44 up:resolve seq 2
  101.  
  102. -918> 2016-01-14 15:50:16.220122 7f96175cf8c0 0 mon.lts-osd2@-1(probing).osd e10235 crush map has features 1107558400, adjusting msgr requires
  103. -917> 2016-01-14 15:50:16.220132 7f96175cf8c0 0 mon.lts-osd2@-1(probing).osd e10235 crush map has features 1107558400, adjusting msgr requires
  104. -916> 2016-01-14 15:50:16.220135 7f96175cf8c0 0 mon.lts-osd2@-1(probing).osd e10235 crush map has features 1107558400, adjusting msgr requires
  105. -915> 2016-01-14 15:50:16.220142 7f96175cf8c0 0 mon.lts-osd2@-1(probing).osd e10235 crush map has features 1107558400, adjusting msgr requires
  106. -914> 2016-01-14 15:50:16.220583 7f96175cf8c0 1 mon.lts-osd2@-1(probing).paxosservice(auth 5501..5688) refresh upgraded, format 0 -> 1
  107. -913> 2016-01-14 15:50:16.222865 7f96175cf8c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-lts-osd2/keyring
  108. -912> 2016-01-14 15:50:16.222874 7f96175cf8c0 5 asok(0x7f961acf8000) register_command mon_status hook 0x7f961b1cf990
  109. -911> 2016-01-14 15:50:16.222879 7f96175cf8c0 5 asok(0x7f961acf8000) register_command quorum_status hook 0x7f961b1cf990
  110. -910> 2016-01-14 15:50:16.222882 7f96175cf8c0 5 asok(0x7f961acf8000) register_command sync_force hook 0x7f961b1cf990
  111. -909> 2016-01-14 15:50:16.222884 7f96175cf8c0 5 asok(0x7f961acf8000) register_command add_bootstrap_peer_hint hook 0x7f961b1cf990
  112. -908> 2016-01-14 15:50:16.222886 7f96175cf8c0 5 asok(0x7f961acf8000) register_command quorum enter hook 0x7f961b1cf990
  113. -907> 2016-01-14 15:50:16.222888 7f96175cf8c0 5 asok(0x7f961acf8000) register_command quorum exit hook 0x7f961b1cf990
  114. -906> 2016-01-14 15:50:16.222891 7f96175cf8c0 5 asok(0x7f961acf8000) register_command ops hook 0x7f961b1cf990
  115. -905> 2016-01-14 15:50:16.222916 7f96175cf8c0 1 -- 10.5.68.230:6789/0 messenger.start
  116. -904> 2016-01-14 15:50:16.222955 7f96175cf8c0 2 mon.lts-osd2@-1(probing) e2 init
  117. -903> 2016-01-14 15:50:16.223023 7f96175cf8c0 1 accepter.accepter.start
  118. -902> 2016-01-14 15:50:16.223047 7f96175cf8c0 0 mon.lts-osd2@-1(probing) e2 my rank is now 1 (was -1)
  119. -901> 2016-01-14 15:50:16.223054 7f96175cf8c0 1 -- 10.5.68.230:6789/0 mark_down_all
  120. -900> 2016-01-14 15:50:16.223075 7f96175cf8c0 1 -- 10.5.68.230:6789/0 --> mon.0 10.5.68.229:6789/0 -- mon_probe(probe cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-osd2) v6 -- ?+0 0x7f961b1e2500
  121. -899> 2016-01-14 15:50:16.223127 7f96175cf8c0 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- mon_probe(probe cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-osd2) v6 -- ?+0 0x7f961b1e2280
  122. -898> 2016-01-14 15:50:16.223369 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  123. -897> 2016-01-14 15:50:16.223417 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  124. -896> 2016-01-14 15:50:16.223430 7f960c0f6700 0 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault
  125. -895> 2016-01-14 15:50:16.223544 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  126. -894> 2016-01-14 15:50:16.223566 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  127. -893> 2016-01-14 15:50:16.224459 7f960b6f3700 0 -- 10.5.68.230:6789/0 >> 10.5.68.236:6789/0 pipe(0x7f961b307000 sd=13 :49477 s=2 pgs=195 cs=1 l=0 c=0x7f961ac80f20).reader missed message? skipped from seq 0 to 1445553909
  128. -892> 2016-01-14 15:50:16.224489 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553909 ==== mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 )) v6 ==== 561+0+0 (927207654 0 0) 0x7f961b1e2280 con 0x7f961ac80f20
  129. -891> 2016-01-14 15:50:16.224513 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224427, event: header_read, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  130. -890> 2016-01-14 15:50:16.224519 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224430, event: throttled, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  131. -889> 2016-01-14 15:50:16.224526 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224457, event: all_read, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  132. -888> 2016-01-14 15:50:16.224529 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224508, event: dispatched, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  133. -887> 2016-01-14 15:50:16.224535 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224535, event: mon:_ms_dispatch, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  134. -886> 2016-01-14 15:50:16.224540 7f960d8f9700 5 mon.lts-osd2@1(probing) e2 _ms_dispatch setting monitor caps on this connection
  135. -885> 2016-01-14 15:50:16.224548 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.224548, event: mon:dispatch_op, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  136. -884> 2016-01-14 15:50:16.224587 7f960d8f9700 0 log_channel(cluster) log [INF] : mon.lts-osd2 calling new monitor election
  137. -883> 2016-01-14 15:50:16.224593 7f960d8f9700 10 log_client _send_to_monlog to self
  138. -882> 2016-01-14 15:50:16.224594 7f960d8f9700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
  139. -881> 2016-01-14 15:50:16.224596 7f960d8f9700 10 log_client will send 2016-01-14 15:50:16.224592 mon.1 10.5.68.230:6789/0 1 : cluster [INF] mon.lts-osd2 calling new monitor election
  140. -880> 2016-01-14 15:50:16.224606 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9df80 con 0x7f961ac818c0
  141. -879> 2016-01-14 15:50:16.224631 7f960d8f9700 5 mon.lts-osd2@1(electing).elector(0) start -- can i be leader?
  142. -878> 2016-01-14 15:50:16.224744 7f960d8f9700 1 mon.lts-osd2@1(electing).elector(1438) init, last seen epoch 1438
  143. -877> 2016-01-14 15:50:16.225840 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> mon.0 10.5.68.229:6789/0 -- election(cabd1728-2eca-4e18-a581-b4885364e5a4 propose 1439) v5 -- ?+0 0x7f961aca1b80
  144. -876> 2016-01-14 15:50:16.225882 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- election(cabd1728-2eca-4e18-a581-b4885364e5a4 propose 1439) v5 -- ?+0 0x7f961aca2100
  145. -875> 2016-01-14 15:50:16.225906 7f960d8f9700 5 -- op tracker -- seq: 1, time: 2016-01-14 15:50:16.225906, event: done, op: mon_probe(reply cabd1728-2eca-4e18-a581-b4885364e5a4 name lts-mon quorum 1,2 paxos( fc 13454856 lc 13455395 ))
  146. -874> 2016-01-14 15:50:16.225934 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9df80 con 0x7f961ac818c0
  147. -873> 2016-01-14 15:50:16.225943 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.224613, event: header_read, op: log(1 entries)
  148. -872> 2016-01-14 15:50:16.225945 7f960d8f9700 5 -- op tracker -- seq: 2, time: 0.000000, event: throttled, op: log(1 entries)
  149. -871> 2016-01-14 15:50:16.225947 7f960d8f9700 5 -- op tracker -- seq: 2, time: 0.000000, event: all_read, op: log(1 entries)
  150. -870> 2016-01-14 15:50:16.225948 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225941, event: dispatched, op: log(1 entries)
  151. -869> 2016-01-14 15:50:16.225951 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225951, event: mon:_ms_dispatch, op: log(1 entries)
  152. -868> 2016-01-14 15:50:16.225955 7f960d8f9700 5 mon.lts-osd2@1(electing) e2 _ms_dispatch setting monitor caps on this connection
  153. -867> 2016-01-14 15:50:16.225959 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225959, event: mon:dispatch_op, op: log(1 entries)
  154. -866> 2016-01-14 15:50:16.225964 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225964, event: psvc:dispatch, op: log(1 entries)
  155. -865> 2016-01-14 15:50:16.225967 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.225968 lease_expire=0.000000 has v0 lc 13455395
  156. -864> 2016-01-14 15:50:16.225976 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225976, event: logm:wait_for_readable, op: log(1 entries)
  157. -863> 2016-01-14 15:50:16.225978 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225978, event: logm:wait_for_readable/paxos, op: log(1 entries)
  158. -862> 2016-01-14 15:50:16.225980 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.225980 lease_expire=0.000000 has v0 lc 13455395
  159. -861> 2016-01-14 15:50:16.225983 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:16.225983, event: paxos:wait_for_readable, op: log(1 entries)
  160. -860> 2016-01-14 15:50:16.226220 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  161. -859> 2016-01-14 15:50:16.226251 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  162. -858> 2016-01-14 15:50:16.246022 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'get_command_descriptions' '' to 0x7f961ac5c230 returned 2165 bytes
  163. -857> 2016-01-14 15:50:16.252084 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  164. -856> 2016-01-14 15:50:16.252095 7f9611720700 10 log_client _send_to_monlog to self
  165. -855> 2016-01-14 15:50:16.252097 7f9611720700 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
  166. -854> 2016-01-14 15:50:16.252100 7f9611720700 10 log_client will send 2016-01-14 15:50:16.252091 mon.1 10.5.68.230:6789/0 2 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  167. -853> 2016-01-14 15:50:16.252122 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9db00 con 0x7f961ac818c0
  168. -852> 2016-01-14 15:50:16.252191 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  169. -851> 2016-01-14 15:50:16.252196 7f9611720700 10 log_client _send_to_monlog to self
  170. -850> 2016-01-14 15:50:16.252197 7f9611720700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
  171. -849> 2016-01-14 15:50:16.252199 7f9611720700 10 log_client will send 2016-01-14 15:50:16.252196 mon.1 10.5.68.230:6789/0 3 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  172. -848> 2016-01-14 15:50:16.252207 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9eac0 con 0x7f961ac818c0
  173. -847> 2016-01-14 15:50:16.252229 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'mon_status' '' to 0x7f961b1cf990 returned 813 bytes
  174. -846> 2016-01-14 15:50:16.252289 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9db00 con 0x7f961ac818c0
  175. -845> 2016-01-14 15:50:16.252306 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252129, event: header_read, op: log(1 entries)
  176. -844> 2016-01-14 15:50:16.252317 7f960d8f9700 5 -- op tracker -- seq: 3, time: 0.000000, event: throttled, op: log(1 entries)
  177. -843> 2016-01-14 15:50:16.252319 7f960d8f9700 5 -- op tracker -- seq: 3, time: 0.000000, event: all_read, op: log(1 entries)
  178. -842> 2016-01-14 15:50:16.252326 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252302, event: dispatched, op: log(1 entries)
  179. -841> 2016-01-14 15:50:16.252329 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252329, event: mon:_ms_dispatch, op: log(1 entries)
  180. -840> 2016-01-14 15:50:16.252337 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252337, event: mon:dispatch_op, op: log(1 entries)
  181. -839> 2016-01-14 15:50:16.252342 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252342, event: psvc:dispatch, op: log(1 entries)
  182. -838> 2016-01-14 15:50:16.252346 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.252347 lease_expire=0.000000 has v0 lc 13455395
  183. -837> 2016-01-14 15:50:16.252355 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252355, event: logm:wait_for_readable, op: log(1 entries)
  184. -836> 2016-01-14 15:50:16.252369 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252369, event: logm:wait_for_readable/paxos, op: log(1 entries)
  185. -835> 2016-01-14 15:50:16.252373 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.252374 lease_expire=0.000000 has v0 lc 13455395
  186. -834> 2016-01-14 15:50:16.252380 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:16.252380, event: paxos:wait_for_readable, op: log(1 entries)
  187. -833> 2016-01-14 15:50:16.252385 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9eac0 con 0x7f961ac818c0
  188. -832> 2016-01-14 15:50:16.252393 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252211, event: header_read, op: log(1 entries)
  189. -831> 2016-01-14 15:50:16.252395 7f960d8f9700 5 -- op tracker -- seq: 4, time: 0.000000, event: throttled, op: log(1 entries)
  190. -830> 2016-01-14 15:50:16.252400 7f960d8f9700 5 -- op tracker -- seq: 4, time: 0.000000, event: all_read, op: log(1 entries)
  191. -829> 2016-01-14 15:50:16.252401 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252392, event: dispatched, op: log(1 entries)
  192. -828> 2016-01-14 15:50:16.252403 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252403, event: mon:_ms_dispatch, op: log(1 entries)
  193. -827> 2016-01-14 15:50:16.252405 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252405, event: mon:dispatch_op, op: log(1 entries)
  194. -826> 2016-01-14 15:50:16.252407 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252407, event: psvc:dispatch, op: log(1 entries)
  195. -825> 2016-01-14 15:50:16.252423 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.252423 lease_expire=0.000000 has v0 lc 13455395
  196. -824> 2016-01-14 15:50:16.252435 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252435, event: logm:wait_for_readable, op: log(1 entries)
  197. -823> 2016-01-14 15:50:16.252437 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252437, event: logm:wait_for_readable/paxos, op: log(1 entries)
  198. -822> 2016-01-14 15:50:16.252438 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:16.252439 lease_expire=0.000000 has v0 lc 13455395
  199. -821> 2016-01-14 15:50:16.252441 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:16.252441, event: paxos:wait_for_readable, op: log(1 entries)
  200. -820> 2016-01-14 15:50:16.270156 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553910 ==== election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439) v5 ==== 27472+0+0 (1388341366 0 0) 0x7f961aca2100 con 0x7f961ac80f20
  201. -819> 2016-01-14 15:50:16.270198 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.269773, event: header_read, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  202. -818> 2016-01-14 15:50:16.270205 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.269775, event: throttled, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  203. -817> 2016-01-14 15:50:16.270208 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270056, event: all_read, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  204. -816> 2016-01-14 15:50:16.270210 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270191, event: dispatched, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  205. -815> 2016-01-14 15:50:16.270214 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270214, event: mon:_ms_dispatch, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  206. -814> 2016-01-14 15:50:16.270219 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270219, event: mon:dispatch_op, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  207. -813> 2016-01-14 15:50:16.270262 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270261, event: elector:dispatch, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  208. -812> 2016-01-14 15:50:16.270288 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270288, event: elector:handle_ack, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  209. -811> 2016-01-14 15:50:16.270294 7f960d8f9700 5 mon.lts-osd2@1(electing).elector(1439) handle_ack from mon.2
  210. -810> 2016-01-14 15:50:16.270298 7f960d8f9700 5 mon.lts-osd2@1(electing).elector(1439) so far i have {1=72057594037927935,2=72057594037927935}
  211. -809> 2016-01-14 15:50:16.270304 7f960d8f9700 5 -- op tracker -- seq: 5, time: 2016-01-14 15:50:16.270304, event: done, op: election(cabd1728-2eca-4e18-a581-b4885364e5a4 ack 1439)
  212. -808> 2016-01-14 15:50:16.626468 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  213. -807> 2016-01-14 15:50:16.626511 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  214. -806> 2016-01-14 15:50:17.314656 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'get_command_descriptions' '' to 0x7f961ac5c230 returned 2165 bytes
  215. -805> 2016-01-14 15:50:17.321171 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  216. -804> 2016-01-14 15:50:17.321182 7f9611720700 10 log_client _send_to_monlog to self
  217. -803> 2016-01-14 15:50:17.321185 7f9611720700 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
  218. -802> 2016-01-14 15:50:17.321188 7f9611720700 10 log_client will send 2016-01-14 15:50:17.321181 mon.1 10.5.68.230:6789/0 4 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  219. -801> 2016-01-14 15:50:17.321201 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9e1c0 con 0x7f961ac818c0
  220. -800> 2016-01-14 15:50:17.321243 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  221. -799> 2016-01-14 15:50:17.321245 7f9611720700 10 log_client _send_to_monlog to self
  222. -798> 2016-01-14 15:50:17.321246 7f9611720700 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
  223. -797> 2016-01-14 15:50:17.321247 7f9611720700 10 log_client will send 2016-01-14 15:50:17.321244 mon.1 10.5.68.230:6789/0 5 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  224. -796> 2016-01-14 15:50:17.321253 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9e640 con 0x7f961ac818c0
  225. -795> 2016-01-14 15:50:17.321270 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'mon_status' '' to 0x7f961b1cf990 returned 813 bytes
  226. -794> 2016-01-14 15:50:17.321284 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9e1c0 con 0x7f961ac818c0
  227. -793> 2016-01-14 15:50:17.321306 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321207, event: header_read, op: log(1 entries)
  228. -792> 2016-01-14 15:50:17.321315 7f960d8f9700 5 -- op tracker -- seq: 6, time: 0.000000, event: throttled, op: log(1 entries)
  229. -791> 2016-01-14 15:50:17.321317 7f960d8f9700 5 -- op tracker -- seq: 6, time: 0.000000, event: all_read, op: log(1 entries)
  230. -790> 2016-01-14 15:50:17.321318 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321301, event: dispatched, op: log(1 entries)
  231. -789> 2016-01-14 15:50:17.321325 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321325, event: mon:_ms_dispatch, op: log(1 entries)
  232. -788> 2016-01-14 15:50:17.321335 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321335, event: mon:dispatch_op, op: log(1 entries)
  233. -787> 2016-01-14 15:50:17.321343 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321343, event: psvc:dispatch, op: log(1 entries)
  234. -786> 2016-01-14 15:50:17.321348 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:17.321349 lease_expire=0.000000 has v0 lc 13455395
  235. -785> 2016-01-14 15:50:17.321357 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321357, event: logm:wait_for_readable, op: log(1 entries)
  236. -784> 2016-01-14 15:50:17.321364 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321364, event: logm:wait_for_readable/paxos, op: log(1 entries)
  237. -783> 2016-01-14 15:50:17.321366 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:17.321366 lease_expire=0.000000 has v0 lc 13455395
  238. -782> 2016-01-14 15:50:17.321371 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:17.321371, event: paxos:wait_for_readable, op: log(1 entries)
  239. -781> 2016-01-14 15:50:17.321375 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9e640 con 0x7f961ac818c0
  240. -780> 2016-01-14 15:50:17.321383 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321256, event: header_read, op: log(1 entries)
  241. -779> 2016-01-14 15:50:17.321385 7f960d8f9700 5 -- op tracker -- seq: 7, time: 0.000000, event: throttled, op: log(1 entries)
  242. -778> 2016-01-14 15:50:17.321389 7f960d8f9700 5 -- op tracker -- seq: 7, time: 0.000000, event: all_read, op: log(1 entries)
  243. -777> 2016-01-14 15:50:17.321390 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321382, event: dispatched, op: log(1 entries)
  244. -776> 2016-01-14 15:50:17.321392 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321392, event: mon:_ms_dispatch, op: log(1 entries)
  245. -775> 2016-01-14 15:50:17.321398 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321398, event: mon:dispatch_op, op: log(1 entries)
  246. -774> 2016-01-14 15:50:17.321401 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321401, event: psvc:dispatch, op: log(1 entries)
  247. -773> 2016-01-14 15:50:17.321405 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:17.321406 lease_expire=0.000000 has v0 lc 13455395
  248. -772> 2016-01-14 15:50:17.321409 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321409, event: logm:wait_for_readable, op: log(1 entries)
  249. -771> 2016-01-14 15:50:17.321413 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321413, event: logm:wait_for_readable/paxos, op: log(1 entries)
  250. -770> 2016-01-14 15:50:17.321415 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:17.321416 lease_expire=0.000000 has v0 lc 13455395
  251. -769> 2016-01-14 15:50:17.321421 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:17.321421, event: paxos:wait_for_readable, op: log(1 entries)
  252. -768> 2016-01-14 15:50:17.426683 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  253. -767> 2016-01-14 15:50:17.426739 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  254. -766> 2016-01-14 15:50:18.382912 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'get_command_descriptions' '' to 0x7f961ac5c230 returned 2165 bytes
  255. -765> 2016-01-14 15:50:18.389197 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  256. -764> 2016-01-14 15:50:18.389207 7f9611720700 10 log_client _send_to_monlog to self
  257. -763> 2016-01-14 15:50:18.389209 7f9611720700 10 log_client log_queue is 6 last_log 6 sent 5 num 6 unsent 1 sending 1
  258. -762> 2016-01-14 15:50:18.389214 7f9611720700 10 log_client will send 2016-01-14 15:50:18.389205 mon.1 10.5.68.230:6789/0 6 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  259. -761> 2016-01-14 15:50:18.389228 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9e880 con 0x7f961ac818c0
  260. -760> 2016-01-14 15:50:18.389278 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  261. -759> 2016-01-14 15:50:18.389283 7f9611720700 10 log_client _send_to_monlog to self
  262. -758> 2016-01-14 15:50:18.389284 7f9611720700 10 log_client log_queue is 7 last_log 7 sent 6 num 7 unsent 1 sending 1
  263. -757> 2016-01-14 15:50:18.389285 7f9611720700 10 log_client will send 2016-01-14 15:50:18.389282 mon.1 10.5.68.230:6789/0 7 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  264. -756> 2016-01-14 15:50:18.389291 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9f180 con 0x7f961ac818c0
  265. -755> 2016-01-14 15:50:18.389295 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9e880 con 0x7f961ac818c0
  266. -754> 2016-01-14 15:50:18.389308 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'mon_status' '' to 0x7f961b1cf990 returned 813 bytes
  267. -753> 2016-01-14 15:50:18.389318 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389235, event: header_read, op: log(1 entries)
  268. -752> 2016-01-14 15:50:18.389325 7f960d8f9700 5 -- op tracker -- seq: 8, time: 0.000000, event: throttled, op: log(1 entries)
  269. -751> 2016-01-14 15:50:18.389335 7f960d8f9700 5 -- op tracker -- seq: 8, time: 0.000000, event: all_read, op: log(1 entries)
  270. -750> 2016-01-14 15:50:18.389336 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389314, event: dispatched, op: log(1 entries)
  271. -749> 2016-01-14 15:50:18.389343 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389343, event: mon:_ms_dispatch, op: log(1 entries)
  272. -748> 2016-01-14 15:50:18.389352 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389352, event: mon:dispatch_op, op: log(1 entries)
  273. -747> 2016-01-14 15:50:18.389374 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389374, event: psvc:dispatch, op: log(1 entries)
  274. -746> 2016-01-14 15:50:18.389383 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:18.389383 lease_expire=0.000000 has v0 lc 13455395
  275. -745> 2016-01-14 15:50:18.389391 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389391, event: logm:wait_for_readable, op: log(1 entries)
  276. -744> 2016-01-14 15:50:18.389394 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389394, event: logm:wait_for_readable/paxos, op: log(1 entries)
  277. -743> 2016-01-14 15:50:18.389398 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:18.389398 lease_expire=0.000000 has v0 lc 13455395
  278. -742> 2016-01-14 15:50:18.389401 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:18.389401, event: paxos:wait_for_readable, op: log(1 entries)
  279. -741> 2016-01-14 15:50:18.389406 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9f180 con 0x7f961ac818c0
  280. -740> 2016-01-14 15:50:18.389415 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389294, event: header_read, op: log(1 entries)
  281. -739> 2016-01-14 15:50:18.389423 7f960d8f9700 5 -- op tracker -- seq: 9, time: 0.000000, event: throttled, op: log(1 entries)
  282. -738> 2016-01-14 15:50:18.389424 7f960d8f9700 5 -- op tracker -- seq: 9, time: 0.000000, event: all_read, op: log(1 entries)
  283. -737> 2016-01-14 15:50:18.389428 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389413, event: dispatched, op: log(1 entries)
  284. -736> 2016-01-14 15:50:18.389434 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389434, event: mon:_ms_dispatch, op: log(1 entries)
  285. -735> 2016-01-14 15:50:18.389436 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389436, event: mon:dispatch_op, op: log(1 entries)
  286. -734> 2016-01-14 15:50:18.389441 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389441, event: psvc:dispatch, op: log(1 entries)
  287. -733> 2016-01-14 15:50:18.389443 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:18.389443 lease_expire=0.000000 has v0 lc 13455395
  288. -732> 2016-01-14 15:50:18.389448 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389448, event: logm:wait_for_readable, op: log(1 entries)
  289. -731> 2016-01-14 15:50:18.389450 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389450, event: logm:wait_for_readable/paxos, op: log(1 entries)
  290. -730> 2016-01-14 15:50:18.389454 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:18.389454 lease_expire=0.000000 has v0 lc 13455395
  291. -729> 2016-01-14 15:50:18.389457 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:18.389457, event: paxos:wait_for_readable, op: log(1 entries)
  292. -728> 2016-01-14 15:50:19.026941 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).connect error 10.5.68.229:6789/0, (111) Connection refused
  293. -727> 2016-01-14 15:50:19.026984 7f960c0f6700 2 -- 10.5.68.230:6789/0 >> 10.5.68.229:6789/0 pipe(0x7f961b302000 sd=12 :0 s=1 pgs=0 cs=0 l=0 c=0x7f961ac81600).fault (111) Connection refused
  294. -726> 2016-01-14 15:50:19.451922 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'get_command_descriptions' '' to 0x7f961ac5c230 returned 2165 bytes
  295. -725> 2016-01-14 15:50:19.458832 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  296. -724> 2016-01-14 15:50:19.458846 7f9611720700 10 log_client _send_to_monlog to self
  297. -723> 2016-01-14 15:50:19.458858 7f9611720700 10 log_client log_queue is 8 last_log 8 sent 7 num 8 unsent 1 sending 1
  298. -722> 2016-01-14 15:50:19.458862 7f9611720700 10 log_client will send 2016-01-14 15:50:19.458845 mon.1 10.5.68.230:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  299. -721> 2016-01-14 15:50:19.458874 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9f3c0 con 0x7f961ac818c0
  300. -720> 2016-01-14 15:50:19.458933 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  301. -719> 2016-01-14 15:50:19.458938 7f9611720700 10 log_client _send_to_monlog to self
  302. -718> 2016-01-14 15:50:19.458939 7f9611720700 10 log_client log_queue is 9 last_log 9 sent 8 num 9 unsent 1 sending 1
  303. -717> 2016-01-14 15:50:19.458941 7f9611720700 10 log_client will send 2016-01-14 15:50:19.458938 mon.1 10.5.68.230:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  304. -716> 2016-01-14 15:50:19.458946 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9ed00 con 0x7f961ac818c0
  305. -715> 2016-01-14 15:50:19.458947 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9f3c0 con 0x7f961ac818c0
  306. -714> 2016-01-14 15:50:19.458963 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'mon_status' '' to 0x7f961b1cf990 returned 813 bytes
  307. -713> 2016-01-14 15:50:19.458966 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.458880, event: header_read, op: log(1 entries)
  308. -712> 2016-01-14 15:50:19.458971 7f960d8f9700 5 -- op tracker -- seq: 10, time: 0.000000, event: throttled, op: log(1 entries)
  309. -711> 2016-01-14 15:50:19.458975 7f960d8f9700 5 -- op tracker -- seq: 10, time: 0.000000, event: all_read, op: log(1 entries)
  310. -710> 2016-01-14 15:50:19.458977 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.458962, event: dispatched, op: log(1 entries)
  311. -709> 2016-01-14 15:50:19.458981 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.458981, event: mon:_ms_dispatch, op: log(1 entries)
  312. -708> 2016-01-14 15:50:19.458984 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.458984, event: mon:dispatch_op, op: log(1 entries)
  313. -707> 2016-01-14 15:50:19.458987 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.458987, event: psvc:dispatch, op: log(1 entries)
  314. -706> 2016-01-14 15:50:19.458993 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:19.458994 lease_expire=0.000000 has v0 lc 13455395
  315. -705> 2016-01-14 15:50:19.459011 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.459011, event: logm:wait_for_readable, op: log(1 entries)
  316. -704> 2016-01-14 15:50:19.459016 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.459016, event: logm:wait_for_readable/paxos, op: log(1 entries)
  317. -703> 2016-01-14 15:50:19.459020 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:19.459022 lease_expire=0.000000 has v0 lc 13455395
  318. -702> 2016-01-14 15:50:19.459026 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:19.459026, event: paxos:wait_for_readable, op: log(1 entries)
  319. -701> 2016-01-14 15:50:19.459030 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9ed00 con 0x7f961ac818c0
  320. -700> 2016-01-14 15:50:19.459038 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.458949, event: header_read, op: log(1 entries)
  321. -699> 2016-01-14 15:50:19.459040 7f960d8f9700 5 -- op tracker -- seq: 11, time: 0.000000, event: throttled, op: log(1 entries)
  322. -698> 2016-01-14 15:50:19.459043 7f960d8f9700 5 -- op tracker -- seq: 11, time: 0.000000, event: all_read, op: log(1 entries)
  323. -697> 2016-01-14 15:50:19.459045 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459037, event: dispatched, op: log(1 entries)
  324. -696> 2016-01-14 15:50:19.459047 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459047, event: mon:_ms_dispatch, op: log(1 entries)
  325. -695> 2016-01-14 15:50:19.459052 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459052, event: mon:dispatch_op, op: log(1 entries)
  326. -694> 2016-01-14 15:50:19.459055 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459055, event: psvc:dispatch, op: log(1 entries)
  327. -693> 2016-01-14 15:50:19.459059 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:19.459059 lease_expire=0.000000 has v0 lc 13455395
  328. -692> 2016-01-14 15:50:19.459062 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459062, event: logm:wait_for_readable, op: log(1 entries)
  329. -691> 2016-01-14 15:50:19.459067 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459067, event: logm:wait_for_readable/paxos, op: log(1 entries)
  330. -690> 2016-01-14 15:50:19.459069 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:19.459069 lease_expire=0.000000 has v0 lc 13455395
  331. -689> 2016-01-14 15:50:19.459074 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:19.459074, event: paxos:wait_for_readable, op: log(1 entries)
  332. -688> 2016-01-14 15:50:20.521205 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'get_command_descriptions' '' to 0x7f961ac5c230 returned 2165 bytes
  333. -687> 2016-01-14 15:50:20.527285 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  334. -686> 2016-01-14 15:50:20.527298 7f9611720700 10 log_client _send_to_monlog to self
  335. -685> 2016-01-14 15:50:20.527300 7f9611720700 10 log_client log_queue is 10 last_log 10 sent 9 num 10 unsent 1 sending 1
  336. -684> 2016-01-14 15:50:20.527304 7f9611720700 10 log_client will send 2016-01-14 15:50:20.527294 mon.1 10.5.68.230:6789/0 10 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
  337. -683> 2016-01-14 15:50:20.527322 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9ef40 con 0x7f961ac818c0
  338. -682> 2016-01-14 15:50:20.527379 7f9611720700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  339. -681> 2016-01-14 15:50:20.527384 7f9611720700 10 log_client _send_to_monlog to self
  340. -680> 2016-01-14 15:50:20.527385 7f9611720700 10 log_client log_queue is 11 last_log 11 sent 10 num 11 unsent 1 sending 1
  341. -679> 2016-01-14 15:50:20.527387 7f9611720700 10 log_client will send 2016-01-14 15:50:20.527383 mon.1 10.5.68.230:6789/0 11 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
  342. -678> 2016-01-14 15:50:20.527402 7f9611720700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9fa80 con 0x7f961ac818c0
  343. -677> 2016-01-14 15:50:20.527430 7f9611720700 5 asok(0x7f961acf8000) AdminSocket: request 'mon_status' '' to 0x7f961b1cf990 returned 813 bytes
  344. -676> 2016-01-14 15:50:20.527485 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9ef40 con 0x7f961ac818c0
  345. -675> 2016-01-14 15:50:20.527523 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527330, event: header_read, op: log(1 entries)
  346. -674> 2016-01-14 15:50:20.527530 7f960d8f9700 5 -- op tracker -- seq: 12, time: 0.000000, event: throttled, op: log(1 entries)
  347. -673> 2016-01-14 15:50:20.527544 7f960d8f9700 5 -- op tracker -- seq: 12, time: 0.000000, event: all_read, op: log(1 entries)
  348. -672> 2016-01-14 15:50:20.527545 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527517, event: dispatched, op: log(1 entries)
  349. -671> 2016-01-14 15:50:20.527547 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527547, event: mon:_ms_dispatch, op: log(1 entries)
  350. -670> 2016-01-14 15:50:20.527565 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527565, event: mon:dispatch_op, op: log(1 entries)
  351. -669> 2016-01-14 15:50:20.527568 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527568, event: psvc:dispatch, op: log(1 entries)
  352. -668> 2016-01-14 15:50:20.527575 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:20.527575 lease_expire=0.000000 has v0 lc 13455395
  353. -667> 2016-01-14 15:50:20.527584 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527584, event: logm:wait_for_readable, op: log(1 entries)
  354. -666> 2016-01-14 15:50:20.527587 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527587, event: logm:wait_for_readable/paxos, op: log(1 entries)
  355. -665> 2016-01-14 15:50:20.527593 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:20.527593 lease_expire=0.000000 has v0 lc 13455395
  356. -664> 2016-01-14 15:50:20.527596 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:20.527595, event: paxos:wait_for_readable, op: log(1 entries)
  357. -663> 2016-01-14 15:50:20.527604 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9fa80 con 0x7f961ac818c0
  358. -662> 2016-01-14 15:50:20.527614 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527419, event: header_read, op: log(1 entries)
  359. -661> 2016-01-14 15:50:20.527622 7f960d8f9700 5 -- op tracker -- seq: 13, time: 0.000000, event: throttled, op: log(1 entries)
  360. -660> 2016-01-14 15:50:20.527624 7f960d8f9700 5 -- op tracker -- seq: 13, time: 0.000000, event: all_read, op: log(1 entries)
  361. -659> 2016-01-14 15:50:20.527625 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527612, event: dispatched, op: log(1 entries)
  362. -658> 2016-01-14 15:50:20.527629 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527629, event: mon:_ms_dispatch, op: log(1 entries)
  363. -657> 2016-01-14 15:50:20.527631 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527631, event: mon:dispatch_op, op: log(1 entries)
  364. -656> 2016-01-14 15:50:20.527635 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527635, event: psvc:dispatch, op: log(1 entries)
  365. -655> 2016-01-14 15:50:20.527638 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:20.527638 lease_expire=0.000000 has v0 lc 13455395
  366. -654> 2016-01-14 15:50:20.527643 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527643, event: logm:wait_for_readable, op: log(1 entries)
  367. -653> 2016-01-14 15:50:20.527646 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527646, event: logm:wait_for_readable/paxos, op: log(1 entries)
  368. -652> 2016-01-14 15:50:20.527656 7f960d8f9700 5 mon.lts-osd2@1(electing).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:20.527656 lease_expire=0.000000 has v0 lc 13455395
  369. -651> 2016-01-14 15:50:20.527658 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:20.527658, event: paxos:wait_for_readable, op: log(1 entries)
  370. -650> 2016-01-14 15:50:21.225999 7f960e0fa700 5 mon.lts-osd2@1(electing).elector(1439) election timer expired
  371. -649> 2016-01-14 15:50:21.226828 7f960e0fa700 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- election(cabd1728-2eca-4e18-a581-b4885364e5a4 victory 1440) v5 -- ?+0 0x7f961aca23c0
  372. -648> 2016-01-14 15:50:21.226897 7f960e0fa700 0 log_channel(cluster) log [INF] : mon.lts-osd2@1 won leader election with quorum 1,2
  373. -647> 2016-01-14 15:50:21.226905 7f960e0fa700 10 log_client _send_to_monlog to self
  374. -646> 2016-01-14 15:50:21.226907 7f960e0fa700 10 log_client log_queue is 12 last_log 12 sent 11 num 12 unsent 1 sending 1
  375. -645> 2016-01-14 15:50:21.226910 7f960e0fa700 10 log_client will send 2016-01-14 15:50:21.226903 mon.1 10.5.68.230:6789/0 12 : cluster [INF] mon.lts-osd2@1 won leader election with quorum 1,2
  376. -644> 2016-01-14 15:50:21.226940 7f960e0fa700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961b374480 con 0x7f961ac818c0
  377. -643> 2016-01-14 15:50:21.227009 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961b374480 con 0x7f961ac818c0
  378. -642> 2016-01-14 15:50:21.227648 7f960e0fa700 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- paxos(collect lc 13455395 fc 13454856 pn 72001 opn 0) v3 -- ?+0 0x7f961b38e000
  379. -641> 2016-01-14 15:50:21.227713 7f960e0fa700 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- time_check( ping e 1440 r 1 ) v1 -- ?+0 0x7f961b374000
  380. -640> 2016-01-14 15:50:21.228674 7f960e0fa700 0 log_channel(cluster) log [INF] : HEALTH_WARN; mds cluster is degraded; 1 mons down, quorum 1,2 lts-osd2,lts-mon
  381. -639> 2016-01-14 15:50:21.228686 7f960e0fa700 10 log_client _send_to_monlog to self
  382. -638> 2016-01-14 15:50:21.228687 7f960e0fa700 10 log_client log_queue is 13 last_log 13 sent 12 num 13 unsent 1 sending 1
  383. -637> 2016-01-14 15:50:21.228689 7f960e0fa700 10 log_client will send 2016-01-14 15:50:21.228685 mon.1 10.5.68.230:6789/0 13 : cluster [INF] HEALTH_WARN; mds cluster is degraded; 1 mons down, quorum 1,2 lts-osd2,lts-mon
  384. -636> 2016-01-14 15:50:21.228697 7f960e0fa700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961b374b40 con 0x7f961ac818c0
  385. -635> 2016-01-14 15:50:21.280400 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.226946, event: header_read, op: log(1 entries)
  386. -634> 2016-01-14 15:50:21.280428 7f960d8f9700 5 -- op tracker -- seq: 14, time: 0.000000, event: throttled, op: log(1 entries)
  387. -633> 2016-01-14 15:50:21.280430 7f960d8f9700 5 -- op tracker -- seq: 14, time: 0.000000, event: all_read, op: log(1 entries)
  388. -632> 2016-01-14 15:50:21.280439 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.227026, event: dispatched, op: log(1 entries)
  389. -631> 2016-01-14 15:50:21.280447 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280447, event: mon:_ms_dispatch, op: log(1 entries)
  390. -630> 2016-01-14 15:50:21.280451 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280451, event: mon:dispatch_op, op: log(1 entries)
  391. -629> 2016-01-14 15:50:21.280465 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280465, event: psvc:dispatch, op: log(1 entries)
  392. -628> 2016-01-14 15:50:21.280472 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.280473 lease_expire=0.000000 has v0 lc 13455395
  393. -627> 2016-01-14 15:50:21.280486 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280486, event: logm:wait_for_readable, op: log(1 entries)
  394. -626> 2016-01-14 15:50:21.280489 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280489, event: logm:wait_for_readable/paxos, op: log(1 entries)
  395. -625> 2016-01-14 15:50:21.280496 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.280498 lease_expire=0.000000 has v0 lc 13455395
  396. -624> 2016-01-14 15:50:21.280506 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.280505, event: paxos:wait_for_readable, op: log(1 entries)
  397. -623> 2016-01-14 15:50:21.280518 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961b374b40 con 0x7f961ac818c0
  398. -622> 2016-01-14 15:50:21.280535 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.228701, event: header_read, op: log(1 entries)
  399. -621> 2016-01-14 15:50:21.280541 7f960d8f9700 5 -- op tracker -- seq: 15, time: 0.000000, event: throttled, op: log(1 entries)
  400. -620> 2016-01-14 15:50:21.280542 7f960d8f9700 5 -- op tracker -- seq: 15, time: 0.000000, event: all_read, op: log(1 entries)
  401. -619> 2016-01-14 15:50:21.280543 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280534, event: dispatched, op: log(1 entries)
  402. -618> 2016-01-14 15:50:21.280547 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280547, event: mon:_ms_dispatch, op: log(1 entries)
  403. -617> 2016-01-14 15:50:21.280552 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280552, event: mon:dispatch_op, op: log(1 entries)
  404. -616> 2016-01-14 15:50:21.280554 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280554, event: psvc:dispatch, op: log(1 entries)
  405. -615> 2016-01-14 15:50:21.280559 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.280559 lease_expire=0.000000 has v0 lc 13455395
  406. -614> 2016-01-14 15:50:21.280562 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280562, event: logm:wait_for_readable, op: log(1 entries)
  407. -613> 2016-01-14 15:50:21.280567 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280567, event: logm:wait_for_readable/paxos, op: log(1 entries)
  408. -612> 2016-01-14 15:50:21.280569 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.280569 lease_expire=0.000000 has v0 lc 13455395
  409. -611> 2016-01-14 15:50:21.280574 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.280574, event: paxos:wait_for_readable, op: log(1 entries)
  410. -610> 2016-01-14 15:50:21.294981 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553911 ==== forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader v3 ==== 345+0+0 (2440994056 0 0) 0x7f961b1fa600 con 0x7f961ac80f20
  411. -609> 2016-01-14 15:50:21.295015 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.294784, event: header_read, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  412. -608> 2016-01-14 15:50:21.295024 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.294788, event: throttled, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  413. -607> 2016-01-14 15:50:21.295038 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.294897, event: all_read, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  414. -606> 2016-01-14 15:50:21.295043 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.295012, event: dispatched, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  415. -605> 2016-01-14 15:50:21.295053 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.295052, event: mon:_ms_dispatch, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  416. -604> 2016-01-14 15:50:21.295057 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.295057, event: mon:dispatch_op, op: forward(mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1 caps allow * tid 476 con_features 72057594037927935) to leader
  417. -603> 2016-01-14 15:50:21.295075 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.294784, event: header_read, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  418. -602> 2016-01-14 15:50:21.295081 7f960d8f9700 5 -- op tracker -- seq: 17, time: 0.000000, event: throttled, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  419. -601> 2016-01-14 15:50:21.295082 7f960d8f9700 5 -- op tracker -- seq: 17, time: 0.000000, event: all_read, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  420. -600> 2016-01-14 15:50:21.295083 7f960d8f9700 5 -- op tracker -- seq: 17, time: 0.000000, event: dispatched, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  421. -599> 2016-01-14 15:50:21.295092 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295092, event: mon:_ms_dispatch, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  422. -598> 2016-01-14 15:50:21.295097 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295096, event: mon:dispatch_op, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  423. -597> 2016-01-14 15:50:21.295158 7f960d8f9700 0 mon.lts-osd2@1(leader) e2 handle_command mon_command({"prefix": "mds setmap", "epoch": 7821} v 0) v1
  424. -596> 2016-01-14 15:50:21.295198 7f960d8f9700 0 log_channel(audit) log [INF] : from='client.614112 :/0' entity='unknown.' cmd=[{"prefix": "mds setmap", "epoch": 7821}]: dispatch
  425. -595> 2016-01-14 15:50:21.295219 7f960d8f9700 10 log_client _send_to_monlog to self
  426. -594> 2016-01-14 15:50:21.295220 7f960d8f9700 10 log_client log_queue is 14 last_log 14 sent 13 num 14 unsent 1 sending 1
  427. -593> 2016-01-14 15:50:21.295223 7f960d8f9700 10 log_client will send 2016-01-14 15:50:21.295217 mon.1 10.5.68.230:6789/0 14 : audit [INF] from='client.614112 :/0' entity='unknown.' cmd=[{"prefix": "mds setmap", "epoch": 7821}]: dispatch
  428. -592> 2016-01-14 15:50:21.295235 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9dd40 con 0x7f961ac818c0
  429. -591> 2016-01-14 15:50:21.295257 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295257, event: psvc:dispatch, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  430. -590> 2016-01-14 15:50:21.295263 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295264 lease_expire=0.000000 has v0 lc 13455395
  431. -589> 2016-01-14 15:50:21.295270 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295270, event: mdsmap:wait_for_readable, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  432. -588> 2016-01-14 15:50:21.295272 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295272, event: mdsmap:wait_for_readable/paxos, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  433. -587> 2016-01-14 15:50:21.295276 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295277 lease_expire=0.000000 has v0 lc 13455395
  434. -586> 2016-01-14 15:50:21.295279 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.295279, event: paxos:wait_for_readable, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  435. -585> 2016-01-14 15:50:21.295287 7f960d8f9700 5 -- op tracker -- seq: 16, time: 2016-01-14 15:50:21.295287, event: done, op: forward(??? ) to leader
  436. -584> 2016-01-14 15:50:21.295300 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553912 ==== forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader v3 ==== 607+0+0 (336135527 0 0) 0x7f961b1fac00 con 0x7f961ac80f20
  437. -583> 2016-01-14 15:50:21.295309 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.294939, event: header_read, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  438. -582> 2016-01-14 15:50:21.295312 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.294939, event: throttled, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  439. -581> 2016-01-14 15:50:21.295317 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.294981, event: all_read, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  440. -580> 2016-01-14 15:50:21.295319 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.295308, event: dispatched, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  441. -579> 2016-01-14 15:50:21.295324 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.295324, event: mon:_ms_dispatch, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  442. -578> 2016-01-14 15:50:21.295327 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.295327, event: mon:dispatch_op, op: forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader
  443. -577> 2016-01-14 15:50:21.295335 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.294939, event: header_read, op: log(1 entries)
  444. -576> 2016-01-14 15:50:21.295339 7f960d8f9700 5 -- op tracker -- seq: 19, time: 0.000000, event: throttled, op: log(1 entries)
  445. -575> 2016-01-14 15:50:21.295340 7f960d8f9700 5 -- op tracker -- seq: 19, time: 0.000000, event: all_read, op: log(1 entries)
  446. -574> 2016-01-14 15:50:21.295341 7f960d8f9700 5 -- op tracker -- seq: 19, time: 0.000000, event: dispatched, op: log(1 entries)
  447. -573> 2016-01-14 15:50:21.295343 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295343, event: mon:_ms_dispatch, op: log(1 entries)
  448. -572> 2016-01-14 15:50:21.295347 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295347, event: mon:dispatch_op, op: log(1 entries)
  449. -571> 2016-01-14 15:50:21.295350 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295350, event: psvc:dispatch, op: log(1 entries)
  450. -570> 2016-01-14 15:50:21.295354 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295355 lease_expire=0.000000 has v0 lc 13455395
  451. -569> 2016-01-14 15:50:21.295358 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295358, event: logm:wait_for_readable, op: log(1 entries)
  452. -568> 2016-01-14 15:50:21.295362 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295362, event: logm:wait_for_readable/paxos, op: log(1 entries)
  453. -567> 2016-01-14 15:50:21.295366 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295366 lease_expire=0.000000 has v0 lc 13455395
  454. -566> 2016-01-14 15:50:21.295371 7f960d8f9700 5 -- op tracker -- seq: 19, time: 2016-01-14 15:50:21.295371, event: paxos:wait_for_readable, op: log(1 entries)
  455. -565> 2016-01-14 15:50:21.295374 7f960d8f9700 5 -- op tracker -- seq: 18, time: 2016-01-14 15:50:21.295374, event: done, op: forward(??? ) to leader
  456. -564> 2016-01-14 15:50:21.295379 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.1 10.5.68.230:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x7f961ac9dd40 con 0x7f961ac818c0
  457. -563> 2016-01-14 15:50:21.295387 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295243, event: header_read, op: log(1 entries)
  458. -562> 2016-01-14 15:50:21.295389 7f960d8f9700 5 -- op tracker -- seq: 20, time: 0.000000, event: throttled, op: log(1 entries)
  459. -561> 2016-01-14 15:50:21.295394 7f960d8f9700 5 -- op tracker -- seq: 20, time: 0.000000, event: all_read, op: log(1 entries)
  460. -560> 2016-01-14 15:50:21.295395 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295386, event: dispatched, op: log(1 entries)
  461. -559> 2016-01-14 15:50:21.295399 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295399, event: mon:_ms_dispatch, op: log(1 entries)
  462. -558> 2016-01-14 15:50:21.295402 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295402, event: mon:dispatch_op, op: log(1 entries)
  463. -557> 2016-01-14 15:50:21.295406 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295406, event: psvc:dispatch, op: log(1 entries)
  464. -556> 2016-01-14 15:50:21.295408 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295408 lease_expire=0.000000 has v0 lc 13455395
  465. -555> 2016-01-14 15:50:21.295413 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295413, event: logm:wait_for_readable, op: log(1 entries)
  466. -554> 2016-01-14 15:50:21.295416 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295415, event: logm:wait_for_readable/paxos, op: log(1 entries)
  467. -553> 2016-01-14 15:50:21.295420 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295420 lease_expire=0.000000 has v0 lc 13455395
  468. -552> 2016-01-14 15:50:21.295422 7f960d8f9700 5 -- op tracker -- seq: 20, time: 2016-01-14 15:50:21.295422, event: paxos:wait_for_readable, op: log(1 entries)
  469. -551> 2016-01-14 15:50:21.295428 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553916 ==== forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader v3 ==== 526+0+0 (103659108 0 0) 0x7f961b1fbb00 con 0x7f961ac80f20
  470. -550> 2016-01-14 15:50:21.295437 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295108, event: header_read, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  471. -549> 2016-01-14 15:50:21.295439 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295108, event: throttled, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  472. -548> 2016-01-14 15:50:21.295444 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295134, event: all_read, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  473. -547> 2016-01-14 15:50:21.295451 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295436, event: dispatched, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  474. -546> 2016-01-14 15:50:21.295453 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295453, event: mon:_ms_dispatch, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  475. -545> 2016-01-14 15:50:21.295458 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295458, event: mon:dispatch_op, op: forward(log(1 entries) v1 caps allow * tid 481 con_features 0) to leader
  476. -544> 2016-01-14 15:50:21.295463 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295108, event: header_read, op: log(1 entries)
  477. -543> 2016-01-14 15:50:21.295467 7f960d8f9700 5 -- op tracker -- seq: 22, time: 0.000000, event: throttled, op: log(1 entries)
  478. -542> 2016-01-14 15:50:21.295468 7f960d8f9700 5 -- op tracker -- seq: 22, time: 0.000000, event: all_read, op: log(1 entries)
  479. -541> 2016-01-14 15:50:21.295470 7f960d8f9700 5 -- op tracker -- seq: 22, time: 0.000000, event: dispatched, op: log(1 entries)
  480. -540> 2016-01-14 15:50:21.295471 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295471, event: mon:_ms_dispatch, op: log(1 entries)
  481. -539> 2016-01-14 15:50:21.295475 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295475, event: mon:dispatch_op, op: log(1 entries)
  482. -538> 2016-01-14 15:50:21.295480 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295480, event: psvc:dispatch, op: log(1 entries)
  483. -537> 2016-01-14 15:50:21.295484 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295485 lease_expire=0.000000 has v0 lc 13455395
  484. -536> 2016-01-14 15:50:21.295487 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295487, event: logm:wait_for_readable, op: log(1 entries)
  485. -535> 2016-01-14 15:50:21.295492 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295492, event: logm:wait_for_readable/paxos, op: log(1 entries)
  486. -534> 2016-01-14 15:50:21.295494 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295494 lease_expire=0.000000 has v0 lc 13455395
  487. -533> 2016-01-14 15:50:21.295498 7f960d8f9700 5 -- op tracker -- seq: 22, time: 2016-01-14 15:50:21.295498, event: paxos:wait_for_readable, op: log(1 entries)
  488. -532> 2016-01-14 15:50:21.295504 7f960d8f9700 5 -- op tracker -- seq: 21, time: 2016-01-14 15:50:21.295504, event: done, op: forward(??? ) to leader
  489. -531> 2016-01-14 15:50:21.295511 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553913 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader v3 ==== 634+0+0 (2077265384 0 0) 0x7f961b1fa900 con 0x7f961ac80f20
  490. -530> 2016-01-14 15:50:21.295524 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.294990, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  491. -529> 2016-01-14 15:50:21.295527 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.294990, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  492. -528> 2016-01-14 15:50:21.295533 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.295037, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  493. -527> 2016-01-14 15:50:21.295536 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.295523, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  494. -526> 2016-01-14 15:50:21.295542 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.295542, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  495. -525> 2016-01-14 15:50:21.295545 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.295545, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932) v4 caps allow profile mds tid 478 con_features 72057594037927935) to leader
  496. -524> 2016-01-14 15:50:21.295554 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.294990, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  497. -523> 2016-01-14 15:50:21.295559 7f960d8f9700 5 -- op tracker -- seq: 24, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  498. -522> 2016-01-14 15:50:21.295570 7f960d8f9700 5 -- op tracker -- seq: 24, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  499. -521> 2016-01-14 15:50:21.295571 7f960d8f9700 5 -- op tracker -- seq: 24, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  500. -520> 2016-01-14 15:50:21.295573 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295573, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  501. -519> 2016-01-14 15:50:21.295578 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295578, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  502. -518> 2016-01-14 15:50:21.295580 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295580, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  503. -517> 2016-01-14 15:50:21.295585 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295585 lease_expire=0.000000 has v0 lc 13455395
  504. -516> 2016-01-14 15:50:21.295588 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295588, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  505. -515> 2016-01-14 15:50:21.295595 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295595, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  506. -514> 2016-01-14 15:50:21.295597 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295597 lease_expire=0.000000 has v0 lc 13455395
  507. -513> 2016-01-14 15:50:21.295602 7f960d8f9700 5 -- op tracker -- seq: 24, time: 2016-01-14 15:50:21.295602, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2934 v7932)
  508. -512> 2016-01-14 15:50:21.295605 7f960d8f9700 5 -- op tracker -- seq: 23, time: 2016-01-14 15:50:21.295605, event: done, op: forward(??? ) to leader
  509. -511> 2016-01-14 15:50:21.295614 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553914 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader v3 ==== 634+0+0 (3255478380 0 0) 0x7f961b1fb200 con 0x7f961ac80f20
  510. -510> 2016-01-14 15:50:21.295624 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295042, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  511. -509> 2016-01-14 15:50:21.295630 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295042, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  512. -508> 2016-01-14 15:50:21.295633 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295070, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  513. -507> 2016-01-14 15:50:21.295638 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295623, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  514. -506> 2016-01-14 15:50:21.295642 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295642, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  515. -505> 2016-01-14 15:50:21.295647 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295647, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932) v4 caps allow profile mds tid 479 con_features 72057594037927935) to leader
  516. -504> 2016-01-14 15:50:21.295652 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295042, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  517. -503> 2016-01-14 15:50:21.295657 7f960d8f9700 5 -- op tracker -- seq: 26, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  518. -502> 2016-01-14 15:50:21.295658 7f960d8f9700 5 -- op tracker -- seq: 26, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  519. -501> 2016-01-14 15:50:21.295659 7f960d8f9700 5 -- op tracker -- seq: 26, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  520. -500> 2016-01-14 15:50:21.295661 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295661, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  521. -499> 2016-01-14 15:50:21.295665 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295665, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  522. -498> 2016-01-14 15:50:21.295668 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295668, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  523. -497> 2016-01-14 15:50:21.295672 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295673 lease_expire=0.000000 has v0 lc 13455395
  524. -496> 2016-01-14 15:50:21.295676 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295676, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  525. -495> 2016-01-14 15:50:21.295680 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295680, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  526. -494> 2016-01-14 15:50:21.295682 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295682 lease_expire=0.000000 has v0 lc 13455395
  527. -493> 2016-01-14 15:50:21.295687 7f960d8f9700 5 -- op tracker -- seq: 26, time: 2016-01-14 15:50:21.295687, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2935 v7932)
  528. -492> 2016-01-14 15:50:21.295690 7f960d8f9700 5 -- op tracker -- seq: 25, time: 2016-01-14 15:50:21.295690, event: done, op: forward(??? ) to leader
  529. -491> 2016-01-14 15:50:21.295697 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553915 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader v3 ==== 634+0+0 (34339577 0 0) 0x7f961b1fc100 con 0x7f961ac80f20
  530. -490> 2016-01-14 15:50:21.295706 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295076, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  531. -489> 2016-01-14 15:50:21.295710 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295076, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  532. -488> 2016-01-14 15:50:21.295715 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295103, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  533. -487> 2016-01-14 15:50:21.295718 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295706, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  534. -486> 2016-01-14 15:50:21.295723 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295723, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  535. -485> 2016-01-14 15:50:21.295730 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295730, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932) v4 caps allow profile mds tid 480 con_features 72057594037927935) to leader
  536. -484> 2016-01-14 15:50:21.295738 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295076, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  537. -483> 2016-01-14 15:50:21.295743 7f960d8f9700 5 -- op tracker -- seq: 28, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  538. -482> 2016-01-14 15:50:21.295744 7f960d8f9700 5 -- op tracker -- seq: 28, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  539. -481> 2016-01-14 15:50:21.295746 7f960d8f9700 5 -- op tracker -- seq: 28, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  540. -480> 2016-01-14 15:50:21.295747 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295747, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  541. -479> 2016-01-14 15:50:21.295753 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295753, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  542. -478> 2016-01-14 15:50:21.295755 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295755, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  543. -477> 2016-01-14 15:50:21.295760 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295760 lease_expire=0.000000 has v0 lc 13455395
  544. -476> 2016-01-14 15:50:21.295763 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295763, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  545. -475> 2016-01-14 15:50:21.295767 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295767, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  546. -474> 2016-01-14 15:50:21.295769 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295770 lease_expire=0.000000 has v0 lc 13455395
  547. -473> 2016-01-14 15:50:21.295774 7f960d8f9700 5 -- op tracker -- seq: 28, time: 2016-01-14 15:50:21.295774, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2936 v7932)
  548. -472> 2016-01-14 15:50:21.295780 7f960d8f9700 5 -- op tracker -- seq: 27, time: 2016-01-14 15:50:21.295780, event: done, op: forward(??? ) to leader
  549. -471> 2016-01-14 15:50:21.295786 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553917 ==== forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader v3 ==== 586+0+0 (1846463817 0 0) 0x7f961b1fb800 con 0x7f961ac80f20
  550. -470> 2016-01-14 15:50:21.295796 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295139, event: header_read, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  551. -469> 2016-01-14 15:50:21.295802 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295140, event: throttled, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  552. -468> 2016-01-14 15:50:21.295804 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295177, event: all_read, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  553. -467> 2016-01-14 15:50:21.295809 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295795, event: dispatched, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  554. -466> 2016-01-14 15:50:21.295812 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295812, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  555. -465> 2016-01-14 15:50:21.295818 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295818, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 462 v 0) v1 caps allow profile osd tid 482 con_features 72057594037927935) to leader
  556. -464> 2016-01-14 15:50:21.295823 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295139, event: header_read, op: pg_stats(0 pgs tid 462 v 0)
  557. -463> 2016-01-14 15:50:21.295828 7f960d8f9700 5 -- op tracker -- seq: 30, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 462 v 0)
  558. -462> 2016-01-14 15:50:21.295829 7f960d8f9700 5 -- op tracker -- seq: 30, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 462 v 0)
  559. -461> 2016-01-14 15:50:21.295830 7f960d8f9700 5 -- op tracker -- seq: 30, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 462 v 0)
  560. -460> 2016-01-14 15:50:21.295834 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295833, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 462 v 0)
  561. -459> 2016-01-14 15:50:21.295838 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295838, event: mon:dispatch_op, op: pg_stats(0 pgs tid 462 v 0)
  562. -458> 2016-01-14 15:50:21.295840 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295840, event: psvc:dispatch, op: pg_stats(0 pgs tid 462 v 0)
  563. -457> 2016-01-14 15:50:21.295845 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295845 lease_expire=0.000000 has v0 lc 13455395
  564. -456> 2016-01-14 15:50:21.295848 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295848, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 462 v 0)
  565. -455> 2016-01-14 15:50:21.295853 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295853, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 462 v 0)
  566. -454> 2016-01-14 15:50:21.295855 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295855 lease_expire=0.000000 has v0 lc 13455395
  567. -453> 2016-01-14 15:50:21.295860 7f960d8f9700 5 -- op tracker -- seq: 30, time: 2016-01-14 15:50:21.295860, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 462 v 0)
  568. -452> 2016-01-14 15:50:21.295865 7f960d8f9700 5 -- op tracker -- seq: 29, time: 2016-01-14 15:50:21.295865, event: done, op: forward(??? ) to leader
  569. -451> 2016-01-14 15:50:21.295872 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553918 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader v3 ==== 634+0+0 (456603867 0 0) 0x7f961b1fd000 con 0x7f961ac80f20
  570. -450> 2016-01-14 15:50:21.295881 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295182, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  571. -449> 2016-01-14 15:50:21.295886 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295183, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  572. -448> 2016-01-14 15:50:21.295894 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295210, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  573. -447> 2016-01-14 15:50:21.295897 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295880, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  574. -446> 2016-01-14 15:50:21.295902 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295902, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  575. -445> 2016-01-14 15:50:21.295905 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295905, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932) v4 caps allow profile mds tid 483 con_features 72057594037927935) to leader
  576. -444> 2016-01-14 15:50:21.295926 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295182, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  577. -443> 2016-01-14 15:50:21.295931 7f960d8f9700 5 -- op tracker -- seq: 32, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  578. -442> 2016-01-14 15:50:21.295933 7f960d8f9700 5 -- op tracker -- seq: 32, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  579. -441> 2016-01-14 15:50:21.295934 7f960d8f9700 5 -- op tracker -- seq: 32, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  580. -440> 2016-01-14 15:50:21.295938 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295938, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  581. -439> 2016-01-14 15:50:21.295941 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295941, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  582. -438> 2016-01-14 15:50:21.295946 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295946, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  583. -437> 2016-01-14 15:50:21.295948 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295948 lease_expire=0.000000 has v0 lc 13455395
  584. -436> 2016-01-14 15:50:21.295953 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295953, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  585. -435> 2016-01-14 15:50:21.295956 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295956, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  586. -434> 2016-01-14 15:50:21.295960 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.295960 lease_expire=0.000000 has v0 lc 13455395
  587. -433> 2016-01-14 15:50:21.295971 7f960d8f9700 5 -- op tracker -- seq: 32, time: 2016-01-14 15:50:21.295971, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2937 v7932)
  588. -432> 2016-01-14 15:50:21.295974 7f960d8f9700 5 -- op tracker -- seq: 31, time: 2016-01-14 15:50:21.295974, event: done, op: forward(??? ) to leader
  589. -431> 2016-01-14 15:50:21.295981 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553919 ==== forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader v3 ==== 586+0+0 (1800858677 0 0) 0x7f961b1fca00 con 0x7f961ac80f20
  590. -430> 2016-01-14 15:50:21.295990 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.295215, event: header_read, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  591. -429> 2016-01-14 15:50:21.295997 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.295215, event: throttled, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  592. -428> 2016-01-14 15:50:21.296002 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.295244, event: all_read, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  593. -427> 2016-01-14 15:50:21.296005 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.295989, event: dispatched, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  594. -426> 2016-01-14 15:50:21.296010 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.296010, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  595. -425> 2016-01-14 15:50:21.296013 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.296013, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 383 v 0) v1 caps allow profile osd tid 484 con_features 72057594037927935) to leader
  596. -424> 2016-01-14 15:50:21.296021 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.295215, event: header_read, op: pg_stats(0 pgs tid 383 v 0)
  597. -423> 2016-01-14 15:50:21.296026 7f960d8f9700 5 -- op tracker -- seq: 34, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 383 v 0)
  598. -422> 2016-01-14 15:50:21.296027 7f960d8f9700 5 -- op tracker -- seq: 34, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 383 v 0)
  599. -421> 2016-01-14 15:50:21.296029 7f960d8f9700 5 -- op tracker -- seq: 34, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 383 v 0)
  600. -420> 2016-01-14 15:50:21.296032 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296032, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 383 v 0)
  601. -419> 2016-01-14 15:50:21.296036 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296036, event: mon:dispatch_op, op: pg_stats(0 pgs tid 383 v 0)
  602. -418> 2016-01-14 15:50:21.296038 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296038, event: psvc:dispatch, op: pg_stats(0 pgs tid 383 v 0)
  603. -417> 2016-01-14 15:50:21.296043 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296043 lease_expire=0.000000 has v0 lc 13455395
  604. -416> 2016-01-14 15:50:21.296046 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296046, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 383 v 0)
  605. -415> 2016-01-14 15:50:21.296051 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296051, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 383 v 0)
  606. -414> 2016-01-14 15:50:21.296056 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296056 lease_expire=0.000000 has v0 lc 13455395
  607. -413> 2016-01-14 15:50:21.296061 7f960d8f9700 5 -- op tracker -- seq: 34, time: 2016-01-14 15:50:21.296061, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 383 v 0)
  608. -412> 2016-01-14 15:50:21.296063 7f960d8f9700 5 -- op tracker -- seq: 33, time: 2016-01-14 15:50:21.296063, event: done, op: forward(??? ) to leader
  609. -411> 2016-01-14 15:50:21.296069 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553920 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader v3 ==== 634+0+0 (3236424392 0 0) 0x7f961b1fc700 con 0x7f961ac80f20
  610. -410> 2016-01-14 15:50:21.296078 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.295249, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  611. -409> 2016-01-14 15:50:21.296083 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.295249, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  612. -408> 2016-01-14 15:50:21.296088 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.295273, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  613. -407> 2016-01-14 15:50:21.296091 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.296078, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  614. -406> 2016-01-14 15:50:21.296097 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.296097, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  615. -405> 2016-01-14 15:50:21.296100 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.296100, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932) v4 caps allow profile mds tid 485 con_features 72057594037927935) to leader
  616. -404> 2016-01-14 15:50:21.296107 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.295249, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  617. -403> 2016-01-14 15:50:21.296112 7f960d8f9700 5 -- op tracker -- seq: 36, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  618. -402> 2016-01-14 15:50:21.296113 7f960d8f9700 5 -- op tracker -- seq: 36, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  619. -401> 2016-01-14 15:50:21.296114 7f960d8f9700 5 -- op tracker -- seq: 36, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  620. -400> 2016-01-14 15:50:21.296118 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296118, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  621. -399> 2016-01-14 15:50:21.296122 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296122, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  622. -398> 2016-01-14 15:50:21.296125 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296125, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  623. -397> 2016-01-14 15:50:21.296129 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296129 lease_expire=0.000000 has v0 lc 13455395
  624. -396> 2016-01-14 15:50:21.296132 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296132, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  625. -395> 2016-01-14 15:50:21.296139 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296139, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  626. -394> 2016-01-14 15:50:21.296144 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296144 lease_expire=0.000000 has v0 lc 13455395
  627. -393> 2016-01-14 15:50:21.296147 7f960d8f9700 5 -- op tracker -- seq: 36, time: 2016-01-14 15:50:21.296147, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2938 v7932)
  628. -392> 2016-01-14 15:50:21.296151 7f960d8f9700 5 -- op tracker -- seq: 35, time: 2016-01-14 15:50:21.296151, event: done, op: forward(??? ) to leader
  629. -391> 2016-01-14 15:50:21.296156 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553921 ==== forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader v3 ==== 578+0+0 (834145837 0 0) 0x7f961b1fdc00 con 0x7f961ac80f20
  630. -390> 2016-01-14 15:50:21.296165 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.295279, event: header_read, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  631. -389> 2016-01-14 15:50:21.296170 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.295279, event: throttled, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  632. -388> 2016-01-14 15:50:21.296182 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.295302, event: all_read, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  633. -387> 2016-01-14 15:50:21.296185 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.296164, event: dispatched, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  634. -386> 2016-01-14 15:50:21.296191 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.296190, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  635. -385> 2016-01-14 15:50:21.296194 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.296194, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 126 v 0) v1 caps allow profile osd tid 486 con_features 72057594037927935) to leader
  636. -384> 2016-01-14 15:50:21.296201 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.295279, event: header_read, op: pg_stats(0 pgs tid 126 v 0)
  637. -383> 2016-01-14 15:50:21.296206 7f960d8f9700 5 -- op tracker -- seq: 38, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 126 v 0)
  638. -382> 2016-01-14 15:50:21.296207 7f960d8f9700 5 -- op tracker -- seq: 38, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 126 v 0)
  639. -381> 2016-01-14 15:50:21.296209 7f960d8f9700 5 -- op tracker -- seq: 38, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 126 v 0)
  640. -380> 2016-01-14 15:50:21.296212 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296212, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 126 v 0)
  641. -379> 2016-01-14 15:50:21.296216 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296216, event: mon:dispatch_op, op: pg_stats(0 pgs tid 126 v 0)
  642. -378> 2016-01-14 15:50:21.296218 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296218, event: psvc:dispatch, op: pg_stats(0 pgs tid 126 v 0)
  643. -377> 2016-01-14 15:50:21.296224 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296225 lease_expire=0.000000 has v0 lc 13455395
  644. -376> 2016-01-14 15:50:21.296230 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296230, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 126 v 0)
  645. -375> 2016-01-14 15:50:21.296232 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296232, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 126 v 0)
  646. -374> 2016-01-14 15:50:21.296236 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296237 lease_expire=0.000000 has v0 lc 13455395
  647. -373> 2016-01-14 15:50:21.296239 7f960d8f9700 5 -- op tracker -- seq: 38, time: 2016-01-14 15:50:21.296239, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 126 v 0)
  648. -372> 2016-01-14 15:50:21.296244 7f960d8f9700 5 -- op tracker -- seq: 37, time: 2016-01-14 15:50:21.296244, event: done, op: forward(??? ) to leader
  649. -371> 2016-01-14 15:50:21.296248 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553922 ==== forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader v3 ==== 598+0+0 (3286232767 0 0) 0x7f961b1fd900 con 0x7f961ac80f20
  650. -370> 2016-01-14 15:50:21.296263 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.295307, event: header_read, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  651. -369> 2016-01-14 15:50:21.296270 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.295307, event: throttled, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  652. -368> 2016-01-14 15:50:21.296274 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.295331, event: all_read, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  653. -367> 2016-01-14 15:50:21.296277 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.296262, event: dispatched, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  654. -366> 2016-01-14 15:50:21.296290 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.296290, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  655. -365> 2016-01-14 15:50:21.296293 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.296293, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 897 v 0) v1 caps allow profile osd tid 487 con_features 72057594037927935) to leader
  656. -364> 2016-01-14 15:50:21.296300 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.295307, event: header_read, op: pg_stats(0 pgs tid 897 v 0)
  657. -363> 2016-01-14 15:50:21.296304 7f960d8f9700 5 -- op tracker -- seq: 40, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 897 v 0)
  658. -362> 2016-01-14 15:50:21.296305 7f960d8f9700 5 -- op tracker -- seq: 40, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 897 v 0)
  659. -361> 2016-01-14 15:50:21.296307 7f960d8f9700 5 -- op tracker -- seq: 40, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 897 v 0)
  660. -360> 2016-01-14 15:50:21.296308 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296308, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 897 v 0)
  661. -359> 2016-01-14 15:50:21.296312 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296312, event: mon:dispatch_op, op: pg_stats(0 pgs tid 897 v 0)
  662. -358> 2016-01-14 15:50:21.296315 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296314, event: psvc:dispatch, op: pg_stats(0 pgs tid 897 v 0)
  663. -357> 2016-01-14 15:50:21.296319 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296319 lease_expire=0.000000 has v0 lc 13455395
  664. -356> 2016-01-14 15:50:21.296322 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296322, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 897 v 0)
  665. -355> 2016-01-14 15:50:21.296326 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296326, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 897 v 0)
  666. -354> 2016-01-14 15:50:21.296329 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296329 lease_expire=0.000000 has v0 lc 13455395
  667. -353> 2016-01-14 15:50:21.296334 7f960d8f9700 5 -- op tracker -- seq: 40, time: 2016-01-14 15:50:21.296334, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 897 v 0)
  668. -352> 2016-01-14 15:50:21.296339 7f960d8f9700 5 -- op tracker -- seq: 39, time: 2016-01-14 15:50:21.296339, event: done, op: forward(??? ) to leader
  669. -351> 2016-01-14 15:50:21.296345 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553923 ==== forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader v3 ==== 602+0+0 (2113989538 0 0) 0x7f961b1fd600 con 0x7f961ac80f20
  670. -350> 2016-01-14 15:50:21.296354 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.295336, event: header_read, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  671. -349> 2016-01-14 15:50:21.296357 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.295336, event: throttled, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  672. -348> 2016-01-14 15:50:21.296361 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.295367, event: all_read, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  673. -347> 2016-01-14 15:50:21.296364 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.296353, event: dispatched, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  674. -346> 2016-01-14 15:50:21.296369 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.296369, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  675. -345> 2016-01-14 15:50:21.296372 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.296372, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 184 v 0) v1 caps allow profile osd tid 488 con_features 72057594037927935) to leader
  676. -344> 2016-01-14 15:50:21.296381 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.295336, event: header_read, op: pg_stats(0 pgs tid 184 v 0)
  677. -343> 2016-01-14 15:50:21.296385 7f960d8f9700 5 -- op tracker -- seq: 42, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 184 v 0)
  678. -342> 2016-01-14 15:50:21.296387 7f960d8f9700 5 -- op tracker -- seq: 42, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 184 v 0)
  679. -341> 2016-01-14 15:50:21.296388 7f960d8f9700 5 -- op tracker -- seq: 42, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 184 v 0)
  680. -340> 2016-01-14 15:50:21.296389 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296389, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 184 v 0)
  681. -339> 2016-01-14 15:50:21.296394 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296394, event: mon:dispatch_op, op: pg_stats(0 pgs tid 184 v 0)
  682. -338> 2016-01-14 15:50:21.296399 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296399, event: psvc:dispatch, op: pg_stats(0 pgs tid 184 v 0)
  683. -337> 2016-01-14 15:50:21.296403 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296404 lease_expire=0.000000 has v0 lc 13455395
  684. -336> 2016-01-14 15:50:21.296406 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296406, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 184 v 0)
  685. -335> 2016-01-14 15:50:21.296411 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296411, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 184 v 0)
  686. -334> 2016-01-14 15:50:21.296413 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296413 lease_expire=0.000000 has v0 lc 13455395
  687. -333> 2016-01-14 15:50:21.296418 7f960d8f9700 5 -- op tracker -- seq: 42, time: 2016-01-14 15:50:21.296418, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 184 v 0)
  688. -332> 2016-01-14 15:50:21.296424 7f960d8f9700 5 -- op tracker -- seq: 41, time: 2016-01-14 15:50:21.296424, event: done, op: forward(??? ) to leader
  689. -331> 2016-01-14 15:50:21.296431 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553924 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader v3 ==== 634+0+0 (3844896344 0 0) 0x7f961b1fd300 con 0x7f961ac80f20
  690. -330> 2016-01-14 15:50:21.296440 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.295372, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  691. -329> 2016-01-14 15:50:21.296448 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.295373, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  692. -328> 2016-01-14 15:50:21.296451 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.295397, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  693. -327> 2016-01-14 15:50:21.296455 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.296439, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  694. -326> 2016-01-14 15:50:21.296460 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.296460, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  695. -325> 2016-01-14 15:50:21.296463 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.296463, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932) v4 caps allow profile mds tid 489 con_features 72057594037927935) to leader
  696. -324> 2016-01-14 15:50:21.296472 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.295372, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  697. -323> 2016-01-14 15:50:21.296477 7f960d8f9700 5 -- op tracker -- seq: 44, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  698. -322> 2016-01-14 15:50:21.296478 7f960d8f9700 5 -- op tracker -- seq: 44, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  699. -321> 2016-01-14 15:50:21.296480 7f960d8f9700 5 -- op tracker -- seq: 44, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  700. -320> 2016-01-14 15:50:21.296481 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296481, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  701. -319> 2016-01-14 15:50:21.296486 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296486, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  702. -318> 2016-01-14 15:50:21.296488 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296488, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  703. -317> 2016-01-14 15:50:21.296492 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296493 lease_expire=0.000000 has v0 lc 13455395
  704. -316> 2016-01-14 15:50:21.296496 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296495, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  705. -315> 2016-01-14 15:50:21.296500 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296500, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  706. -314> 2016-01-14 15:50:21.296502 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296502 lease_expire=0.000000 has v0 lc 13455395
  707. -313> 2016-01-14 15:50:21.296507 7f960d8f9700 5 -- op tracker -- seq: 44, time: 2016-01-14 15:50:21.296507, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2939 v7932)
  708. -312> 2016-01-14 15:50:21.296510 7f960d8f9700 5 -- op tracker -- seq: 43, time: 2016-01-14 15:50:21.296510, event: done, op: forward(??? ) to leader
  709. -311> 2016-01-14 15:50:21.296516 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553925 ==== forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader v3 ==== 578+0+0 (693030050 0 0) 0x7f961b1fa600 con 0x7f961ac80f20
  710. -310> 2016-01-14 15:50:21.296525 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.295402, event: header_read, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  711. -309> 2016-01-14 15:50:21.296528 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.295403, event: throttled, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  712. -308> 2016-01-14 15:50:21.296533 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.295438, event: all_read, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  713. -307> 2016-01-14 15:50:21.296536 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.296524, event: dispatched, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  714. -306> 2016-01-14 15:50:21.296541 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.296541, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  715. -305> 2016-01-14 15:50:21.296543 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.296543, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 286 v 0) v1 caps allow profile osd tid 490 con_features 72057594037927935) to leader
  716. -304> 2016-01-14 15:50:21.296550 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.295402, event: header_read, op: pg_stats(0 pgs tid 286 v 0)
  717. -303> 2016-01-14 15:50:21.296555 7f960d8f9700 5 -- op tracker -- seq: 46, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 286 v 0)
  718. -302> 2016-01-14 15:50:21.296556 7f960d8f9700 5 -- op tracker -- seq: 46, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 286 v 0)
  719. -301> 2016-01-14 15:50:21.296558 7f960d8f9700 5 -- op tracker -- seq: 46, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 286 v 0)
  720. -300> 2016-01-14 15:50:21.296559 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296559, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 286 v 0)
  721. -299> 2016-01-14 15:50:21.296564 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296563, event: mon:dispatch_op, op: pg_stats(0 pgs tid 286 v 0)
  722. -298> 2016-01-14 15:50:21.296569 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296569, event: psvc:dispatch, op: pg_stats(0 pgs tid 286 v 0)
  723. -297> 2016-01-14 15:50:21.296573 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296573 lease_expire=0.000000 has v0 lc 13455395
  724. -296> 2016-01-14 15:50:21.296576 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296576, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 286 v 0)
  725. -295> 2016-01-14 15:50:21.296580 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296580, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 286 v 0)
  726. -294> 2016-01-14 15:50:21.296582 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296583 lease_expire=0.000000 has v0 lc 13455395
  727. -293> 2016-01-14 15:50:21.296587 7f960d8f9700 5 -- op tracker -- seq: 46, time: 2016-01-14 15:50:21.296587, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 286 v 0)
  728. -292> 2016-01-14 15:50:21.296593 7f960d8f9700 5 -- op tracker -- seq: 45, time: 2016-01-14 15:50:21.296593, event: done, op: forward(??? ) to leader
  729. -291> 2016-01-14 15:50:21.296599 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553926 ==== forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader v3 ==== 590+0+0 (936966183 0 0) 0x7f961b3af500 con 0x7f961ac80f20
  730. -290> 2016-01-14 15:50:21.296607 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.295443, event: header_read, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  731. -289> 2016-01-14 15:50:21.296610 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.295444, event: throttled, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  732. -288> 2016-01-14 15:50:21.296623 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.295477, event: all_read, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  733. -287> 2016-01-14 15:50:21.296625 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.296606, event: dispatched, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  734. -286> 2016-01-14 15:50:21.296631 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.296631, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  735. -285> 2016-01-14 15:50:21.296633 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.296633, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 979 v 0) v1 caps allow profile osd tid 491 con_features 72057594037927935) to leader
  736. -284> 2016-01-14 15:50:21.296640 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.295443, event: header_read, op: pg_stats(0 pgs tid 979 v 0)
  737. -283> 2016-01-14 15:50:21.296645 7f960d8f9700 5 -- op tracker -- seq: 48, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 979 v 0)
  738. -282> 2016-01-14 15:50:21.296646 7f960d8f9700 5 -- op tracker -- seq: 48, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 979 v 0)
  739. -281> 2016-01-14 15:50:21.296647 7f960d8f9700 5 -- op tracker -- seq: 48, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 979 v 0)
  740. -280> 2016-01-14 15:50:21.296649 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296649, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 979 v 0)
  741. -279> 2016-01-14 15:50:21.296653 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296653, event: mon:dispatch_op, op: pg_stats(0 pgs tid 979 v 0)
  742. -278> 2016-01-14 15:50:21.296686 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296686, event: psvc:dispatch, op: pg_stats(0 pgs tid 979 v 0)
  743. -277> 2016-01-14 15:50:21.296689 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296689 lease_expire=0.000000 has v0 lc 13455395
  744. -276> 2016-01-14 15:50:21.296692 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296692, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 979 v 0)
  745. -275> 2016-01-14 15:50:21.296694 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296694, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 979 v 0)
  746. -274> 2016-01-14 15:50:21.296695 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296696 lease_expire=0.000000 has v0 lc 13455395
  747. -273> 2016-01-14 15:50:21.296698 7f960d8f9700 5 -- op tracker -- seq: 48, time: 2016-01-14 15:50:21.296698, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 979 v 0)
  748. -272> 2016-01-14 15:50:21.296700 7f960d8f9700 5 -- op tracker -- seq: 47, time: 2016-01-14 15:50:21.296700, event: done, op: forward(??? ) to leader
  749. -271> 2016-01-14 15:50:21.296705 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553927 ==== forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader v3 ==== 594+0+0 (248700600 0 0) 0x7f961b3af200 con 0x7f961ac80f20
  750. -270> 2016-01-14 15:50:21.296712 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.295482, event: header_read, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  751. -269> 2016-01-14 15:50:21.296714 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.295482, event: throttled, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  752. -268> 2016-01-14 15:50:21.296717 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.295505, event: all_read, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  753. -267> 2016-01-14 15:50:21.296719 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.296711, event: dispatched, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  754. -266> 2016-01-14 15:50:21.296721 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.296721, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  755. -265> 2016-01-14 15:50:21.296724 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.296724, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 161 v 0) v1 caps allow profile osd tid 492 con_features 72057594037927935) to leader
  756. -264> 2016-01-14 15:50:21.296728 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.295482, event: header_read, op: pg_stats(0 pgs tid 161 v 0)
  757. -263> 2016-01-14 15:50:21.296730 7f960d8f9700 5 -- op tracker -- seq: 50, time: 0.000000, event: throttled, op: pg_stats(0 pgs tid 161 v 0)
  758. -262> 2016-01-14 15:50:21.296731 7f960d8f9700 5 -- op tracker -- seq: 50, time: 0.000000, event: all_read, op: pg_stats(0 pgs tid 161 v 0)
  759. -261> 2016-01-14 15:50:21.296732 7f960d8f9700 5 -- op tracker -- seq: 50, time: 0.000000, event: dispatched, op: pg_stats(0 pgs tid 161 v 0)
  760. -260> 2016-01-14 15:50:21.296733 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296733, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 161 v 0)
  761. -259> 2016-01-14 15:50:21.296735 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296735, event: mon:dispatch_op, op: pg_stats(0 pgs tid 161 v 0)
  762. -258> 2016-01-14 15:50:21.296737 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296737, event: psvc:dispatch, op: pg_stats(0 pgs tid 161 v 0)
  763. -257> 2016-01-14 15:50:21.296738 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296739 lease_expire=0.000000 has v0 lc 13455395
  764. -256> 2016-01-14 15:50:21.296742 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296742, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 161 v 0)
  765. -255> 2016-01-14 15:50:21.296744 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296744, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 161 v 0)
  766. -254> 2016-01-14 15:50:21.296745 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296746 lease_expire=0.000000 has v0 lc 13455395
  767. -253> 2016-01-14 15:50:21.296748 7f960d8f9700 5 -- op tracker -- seq: 50, time: 2016-01-14 15:50:21.296748, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 161 v 0)
  768. -252> 2016-01-14 15:50:21.296750 7f960d8f9700 5 -- op tracker -- seq: 49, time: 2016-01-14 15:50:21.296750, event: done, op: forward(??? ) to leader
  769. -251> 2016-01-14 15:50:21.296753 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553928 ==== forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader v3 ==== 1135+0+0 (3129730035 0 0) 0x7f961b3aef00 con 0x7f961ac80f20
  770. -250> 2016-01-14 15:50:21.296759 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.295509, event: header_read, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  771. -249> 2016-01-14 15:50:21.296762 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.295516, event: throttled, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  772. -248> 2016-01-14 15:50:21.296764 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.295564, event: all_read, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  773. -247> 2016-01-14 15:50:21.296766 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.296758, event: dispatched, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  774. -246> 2016-01-14 15:50:21.296768 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.296768, event: mon:_ms_dispatch, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  775. -245> 2016-01-14 15:50:21.296771 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.296771, event: mon:dispatch_op, op: forward(pg_stats(1 pgs tid 501 v 0) v1 caps allow profile osd tid 493 con_features 72057594037927935) to leader
  776. -244> 2016-01-14 15:50:21.296775 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.295509, event: header_read, op: pg_stats(1 pgs tid 501 v 0)
  777. -243> 2016-01-14 15:50:21.296777 7f960d8f9700 5 -- op tracker -- seq: 52, time: 0.000000, event: throttled, op: pg_stats(1 pgs tid 501 v 0)
  778. -242> 2016-01-14 15:50:21.296778 7f960d8f9700 5 -- op tracker -- seq: 52, time: 0.000000, event: all_read, op: pg_stats(1 pgs tid 501 v 0)
  779. -241> 2016-01-14 15:50:21.296779 7f960d8f9700 5 -- op tracker -- seq: 52, time: 0.000000, event: dispatched, op: pg_stats(1 pgs tid 501 v 0)
  780. -240> 2016-01-14 15:50:21.296780 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296780, event: mon:_ms_dispatch, op: pg_stats(1 pgs tid 501 v 0)
  781. -239> 2016-01-14 15:50:21.296782 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296782, event: mon:dispatch_op, op: pg_stats(1 pgs tid 501 v 0)
  782. -238> 2016-01-14 15:50:21.296784 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296784, event: psvc:dispatch, op: pg_stats(1 pgs tid 501 v 0)
  783. -237> 2016-01-14 15:50:21.296786 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296786 lease_expire=0.000000 has v0 lc 13455395
  784. -236> 2016-01-14 15:50:21.296788 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296788, event: pgmap:wait_for_readable, op: pg_stats(1 pgs tid 501 v 0)
  785. -235> 2016-01-14 15:50:21.296790 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296790, event: pgmap:wait_for_readable/paxos, op: pg_stats(1 pgs tid 501 v 0)
  786. -234> 2016-01-14 15:50:21.296792 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296792 lease_expire=0.000000 has v0 lc 13455395
  787. -233> 2016-01-14 15:50:21.296794 7f960d8f9700 5 -- op tracker -- seq: 52, time: 2016-01-14 15:50:21.296794, event: paxos:wait_for_readable, op: pg_stats(1 pgs tid 501 v 0)
  788. -232> 2016-01-14 15:50:21.296796 7f960d8f9700 5 -- op tracker -- seq: 51, time: 2016-01-14 15:50:21.296796, event: done, op: forward(??? ) to leader
  789. -231> 2016-01-14 15:50:21.296800 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553929 ==== forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader v3 ==== 1171+0+0 (1089298598 0 0) 0x7f961b3aec00 con 0x7f961ac80f20
  790. -230> 2016-01-14 15:50:21.296806 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.295570, event: header_read, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  791. -229> 2016-01-14 15:50:21.296809 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.295570, event: throttled, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  792. -228> 2016-01-14 15:50:21.296811 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.295608, event: all_read, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  793. -227> 2016-01-14 15:50:21.296813 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.296805, event: dispatched, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  794. -226> 2016-01-14 15:50:21.296815 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.296815, event: mon:_ms_dispatch, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  795. -225> 2016-01-14 15:50:21.296818 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.296818, event: mon:dispatch_op, op: forward(pg_stats(1 pgs tid 951 v 0) v1 caps allow profile osd tid 494 con_features 72057594037927935) to leader
  796. -224> 2016-01-14 15:50:21.296823 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.295570, event: header_read, op: pg_stats(1 pgs tid 951 v 0)
  797. -223> 2016-01-14 15:50:21.296825 7f960d8f9700 5 -- op tracker -- seq: 54, time: 0.000000, event: throttled, op: pg_stats(1 pgs tid 951 v 0)
  798. -222> 2016-01-14 15:50:21.296826 7f960d8f9700 5 -- op tracker -- seq: 54, time: 0.000000, event: all_read, op: pg_stats(1 pgs tid 951 v 0)
  799. -221> 2016-01-14 15:50:21.296828 7f960d8f9700 5 -- op tracker -- seq: 54, time: 0.000000, event: dispatched, op: pg_stats(1 pgs tid 951 v 0)
  800. -220> 2016-01-14 15:50:21.296829 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296829, event: mon:_ms_dispatch, op: pg_stats(1 pgs tid 951 v 0)
  801. -219> 2016-01-14 15:50:21.296831 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296831, event: mon:dispatch_op, op: pg_stats(1 pgs tid 951 v 0)
  802. -218> 2016-01-14 15:50:21.296833 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296833, event: psvc:dispatch, op: pg_stats(1 pgs tid 951 v 0)
  803. -217> 2016-01-14 15:50:21.296835 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296835 lease_expire=0.000000 has v0 lc 13455395
  804. -216> 2016-01-14 15:50:21.296837 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296837, event: pgmap:wait_for_readable, op: pg_stats(1 pgs tid 951 v 0)
  805. -215> 2016-01-14 15:50:21.296839 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296839, event: pgmap:wait_for_readable/paxos, op: pg_stats(1 pgs tid 951 v 0)
  806. -214> 2016-01-14 15:50:21.296864 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296865 lease_expire=0.000000 has v0 lc 13455395
  807. -213> 2016-01-14 15:50:21.296868 7f960d8f9700 5 -- op tracker -- seq: 54, time: 2016-01-14 15:50:21.296868, event: paxos:wait_for_readable, op: pg_stats(1 pgs tid 951 v 0)
  808. -212> 2016-01-14 15:50:21.296870 7f960d8f9700 5 -- op tracker -- seq: 53, time: 2016-01-14 15:50:21.296870, event: done, op: forward(??? ) to leader
  809. -211> 2016-01-14 15:50:21.296877 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553930 ==== forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader v3 ==== 1187+0+0 (2321704322 0 0) 0x7f961b3ae900 con 0x7f961ac80f20
  810. -210> 2016-01-14 15:50:21.296896 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.295619, event: header_read, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  811. -209> 2016-01-14 15:50:21.296898 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.295619, event: throttled, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  812. -208> 2016-01-14 15:50:21.296901 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.295661, event: all_read, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  813. -207> 2016-01-14 15:50:21.296903 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.296895, event: dispatched, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  814. -206> 2016-01-14 15:50:21.296905 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.296905, event: mon:_ms_dispatch, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  815. -205> 2016-01-14 15:50:21.296908 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.296908, event: mon:dispatch_op, op: forward(pg_stats(1 pgs tid 916 v 0) v1 caps allow profile osd tid 495 con_features 72057594037927935) to leader
  816. -204> 2016-01-14 15:50:21.296918 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.295619, event: header_read, op: pg_stats(1 pgs tid 916 v 0)
  817. -203> 2016-01-14 15:50:21.296926 7f960d8f9700 5 -- op tracker -- seq: 56, time: 0.000000, event: throttled, op: pg_stats(1 pgs tid 916 v 0)
  818. -202> 2016-01-14 15:50:21.296928 7f960d8f9700 5 -- op tracker -- seq: 56, time: 0.000000, event: all_read, op: pg_stats(1 pgs tid 916 v 0)
  819. -201> 2016-01-14 15:50:21.296929 7f960d8f9700 5 -- op tracker -- seq: 56, time: 0.000000, event: dispatched, op: pg_stats(1 pgs tid 916 v 0)
  820. -200> 2016-01-14 15:50:21.296930 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296930, event: mon:_ms_dispatch, op: pg_stats(1 pgs tid 916 v 0)
  821. -199> 2016-01-14 15:50:21.296935 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296935, event: mon:dispatch_op, op: pg_stats(1 pgs tid 916 v 0)
  822. -198> 2016-01-14 15:50:21.296937 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296937, event: psvc:dispatch, op: pg_stats(1 pgs tid 916 v 0)
  823. -197> 2016-01-14 15:50:21.296956 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296957 lease_expire=0.000000 has v0 lc 13455395
  824. -196> 2016-01-14 15:50:21.296971 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296971, event: pgmap:wait_for_readable, op: pg_stats(1 pgs tid 916 v 0)
  825. -195> 2016-01-14 15:50:21.296974 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296974, event: pgmap:wait_for_readable/paxos, op: pg_stats(1 pgs tid 916 v 0)
  826. -194> 2016-01-14 15:50:21.296976 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.296976 lease_expire=0.000000 has v0 lc 13455395
  827. -193> 2016-01-14 15:50:21.296978 7f960d8f9700 5 -- op tracker -- seq: 56, time: 2016-01-14 15:50:21.296978, event: paxos:wait_for_readable, op: pg_stats(1 pgs tid 916 v 0)
  828. -192> 2016-01-14 15:50:21.296981 7f960d8f9700 5 -- op tracker -- seq: 55, time: 2016-01-14 15:50:21.296981, event: done, op: forward(??? ) to leader
  829. -191> 2016-01-14 15:50:21.296985 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553931 ==== forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader v3 ==== 510+0+0 (3991860046 0 0) 0x7f961b3ae600 con 0x7f961ac80f20
  830. -190> 2016-01-14 15:50:21.296995 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.295667, event: header_read, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  831. -189> 2016-01-14 15:50:21.296998 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.295667, event: throttled, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  832. -188> 2016-01-14 15:50:21.297000 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.295691, event: all_read, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  833. -187> 2016-01-14 15:50:21.297002 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.296994, event: dispatched, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  834. -186> 2016-01-14 15:50:21.297004 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.297004, event: mon:_ms_dispatch, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  835. -185> 2016-01-14 15:50:21.297007 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.297007, event: mon:dispatch_op, op: forward(log(1 entries) v1 caps allow profile osd tid 496 con_features 72057594037927935) to leader
  836. -184> 2016-01-14 15:50:21.297011 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.295667, event: header_read, op: log(1 entries)
  837. -183> 2016-01-14 15:50:21.297012 7f960d8f9700 5 -- op tracker -- seq: 58, time: 0.000000, event: throttled, op: log(1 entries)
  838. -182> 2016-01-14 15:50:21.297013 7f960d8f9700 5 -- op tracker -- seq: 58, time: 0.000000, event: all_read, op: log(1 entries)
  839. -181> 2016-01-14 15:50:21.297014 7f960d8f9700 5 -- op tracker -- seq: 58, time: 0.000000, event: dispatched, op: log(1 entries)
  840. -180> 2016-01-14 15:50:21.297015 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297015, event: mon:_ms_dispatch, op: log(1 entries)
  841. -179> 2016-01-14 15:50:21.297017 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297017, event: mon:dispatch_op, op: log(1 entries)
  842. -178> 2016-01-14 15:50:21.297019 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297019, event: psvc:dispatch, op: log(1 entries)
  843. -177> 2016-01-14 15:50:21.297021 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.297021 lease_expire=0.000000 has v0 lc 13455395
  844. -176> 2016-01-14 15:50:21.297023 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297023, event: logm:wait_for_readable, op: log(1 entries)
  845. -175> 2016-01-14 15:50:21.297025 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297025, event: logm:wait_for_readable/paxos, op: log(1 entries)
  846. -174> 2016-01-14 15:50:21.297027 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.297027 lease_expire=0.000000 has v0 lc 13455395
  847. -173> 2016-01-14 15:50:21.297029 7f960d8f9700 5 -- op tracker -- seq: 58, time: 2016-01-14 15:50:21.297029, event: paxos:wait_for_readable, op: log(1 entries)
  848. -172> 2016-01-14 15:50:21.297031 7f960d8f9700 5 -- op tracker -- seq: 57, time: 2016-01-14 15:50:21.297031, event: done, op: forward(??? ) to leader
  849. -171> 2016-01-14 15:50:21.297035 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553932 ==== forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader v3 ==== 634+0+0 (4018396000 0 0) 0x7f961b3ae300 con 0x7f961ac80f20
  850. -170> 2016-01-14 15:50:21.297045 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.295696, event: header_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  851. -169> 2016-01-14 15:50:21.297048 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.295696, event: throttled, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  852. -168> 2016-01-14 15:50:21.297050 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.295719, event: all_read, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  853. -167> 2016-01-14 15:50:21.297052 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.297044, event: dispatched, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  854. -166> 2016-01-14 15:50:21.297055 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.297055, event: mon:_ms_dispatch, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  855. -165> 2016-01-14 15:50:21.297058 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.297057, event: mon:dispatch_op, op: forward(mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932) v4 caps allow profile mds tid 497 con_features 72057594037927935) to leader
  856. -164> 2016-01-14 15:50:21.297070 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.295696, event: header_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  857. -163> 2016-01-14 15:50:21.297073 7f960d8f9700 5 -- op tracker -- seq: 60, time: 0.000000, event: throttled, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  858. -162> 2016-01-14 15:50:21.297074 7f960d8f9700 5 -- op tracker -- seq: 60, time: 0.000000, event: all_read, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  859. -161> 2016-01-14 15:50:21.297075 7f960d8f9700 5 -- op tracker -- seq: 60, time: 0.000000, event: dispatched, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  860. -160> 2016-01-14 15:50:21.297077 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297077, event: mon:_ms_dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  861. -159> 2016-01-14 15:50:21.297079 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297079, event: mon:dispatch_op, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  862. -158> 2016-01-14 15:50:21.297081 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297081, event: psvc:dispatch, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  863. -157> 2016-01-14 15:50:21.297083 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.297083 lease_expire=0.000000 has v0 lc 13455395
  864. -156> 2016-01-14 15:50:21.297085 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297085, event: mdsmap:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  865. -155> 2016-01-14 15:50:21.297087 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297087, event: mdsmap:wait_for_readable/paxos, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  866. -154> 2016-01-14 15:50:21.297090 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos recovering c 13454856..13455395) is_readable = 0 - now=2016-01-14 15:50:21.297173 lease_expire=0.000000 has v0 lc 13455395
  867. -153> 2016-01-14 15:50:21.297182 7f960d8f9700 5 -- op tracker -- seq: 60, time: 2016-01-14 15:50:21.297182, event: paxos:wait_for_readable, op: mdsbeacon(504996/lts-osd1 up:resolve seq 2940 v7932)
  868. -152> 2016-01-14 15:50:21.297186 7f960d8f9700 5 -- op tracker -- seq: 59, time: 2016-01-14 15:50:21.297186, event: done, op: forward(??? ) to leader
  869. -151> 2016-01-14 15:50:21.320046 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553933 ==== mon_metadata magic: 0 v1 ==== 396+0+0 (830242547 0 0) 0x7f961acb3400 con 0x7f961ac80f20
  870. -150> 2016-01-14 15:50:21.320072 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.319946, event: header_read, op: mon_metadata magic: 0
  871. -149> 2016-01-14 15:50:21.320077 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.319948, event: throttled, op: mon_metadata magic: 0
  872. -148> 2016-01-14 15:50:21.320079 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.319984, event: all_read, op: mon_metadata magic: 0
  873. -147> 2016-01-14 15:50:21.320084 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.320069, event: dispatched, op: mon_metadata magic: 0
  874. -146> 2016-01-14 15:50:21.320090 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.320090, event: mon:_ms_dispatch, op: mon_metadata magic: 0
  875. -145> 2016-01-14 15:50:21.320093 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.320093, event: mon:dispatch_op, op: mon_metadata magic: 0
  876. -144> 2016-01-14 15:50:21.320242 7f960d8f9700 5 -- op tracker -- seq: 61, time: 2016-01-14 15:50:21.320242, event: done, op: mon_metadata magic: 0
  877. -143> 2016-01-14 15:50:21.361513 7f960d8f9700 1 -- 10.5.68.230:6789/0 <== mon.2 10.5.68.236:6789/0 1445553934 ==== paxos(last lc 13455395 fc 13454856 pn 72001 opn 0) v3 ==== 80+0+0 (2746114684 0 0) 0x7f961b390800 con 0x7f961ac80f20
  878. -142> 2016-01-14 15:50:21.361540 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361449, event: header_read, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  879. -141> 2016-01-14 15:50:21.361545 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361452, event: throttled, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  880. -140> 2016-01-14 15:50:21.361547 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361481, event: all_read, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  881. -139> 2016-01-14 15:50:21.361554 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361537, event: dispatched, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  882. -138> 2016-01-14 15:50:21.361557 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361557, event: mon:_ms_dispatch, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  883. -137> 2016-01-14 15:50:21.361564 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361564, event: mon:dispatch_op, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  884. -136> 2016-01-14 15:50:21.361577 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361577, event: paxos:dispatch, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  885. -135> 2016-01-14 15:50:21.361584 7f960d8f9700 5 -- op tracker -- seq: 62, time: 2016-01-14 15:50:21.361584, event: paxos:handle_last, op: paxos(last lc 13455395 fc 13454856 pn 72001 opn 0)
  886. -134> 2016-01-14 15:50:21.361645 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> mon.2 10.5.68.236:6789/0 -- paxos(lease lc 13455395 fc 13454856 pn 0 opn 0) v3 -- ?+0 0x7f961b1e2280
  887. -133> 2016-01-14 15:50:21.362013 7f960d8f9700 0 log_channel(cluster) log [INF] : monmap e2: 3 mons at {lts-mon=10.5.68.236:6789/0,lts-osd1=10.5.68.229:6789/0,lts-osd2=10.5.68.230:6789/0}
  888. -132> 2016-01-14 15:50:21.362024 7f960d8f9700 10 log_client _send_to_monlog to self
  889. -131> 2016-01-14 15:50:21.362026 7f960d8f9700 10 log_client log_queue is 15 last_log 15 sent 14 num 15 unsent 1 sending 1
  890. -130> 2016-01-14 15:50:21.362029 7f960d8f9700 10 log_client will send 2016-01-14 15:50:21.362022 mon.1 10.5.68.230:6789/0 15 : cluster [INF] monmap e2: 3 mons at {lts-mon=10.5.68.236:6789/0,lts-osd1=10.5.68.229:6789/0,lts-osd2=10.5.68.230:6789/0}
  891. -129> 2016-01-14 15:50:21.362036 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961b375440 con 0x7f961ac818c0
  892. -128> 2016-01-14 15:50:21.362099 7f960d8f9700 0 log_channel(cluster) log [INF] : pgmap v6715571: 4192 pgs: 4 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 4185 active+clean; 31748 GB data, 96182 GB used, 273 TB / 367 TB avail
  893. -127> 2016-01-14 15:50:21.362104 7f960d8f9700 10 log_client _send_to_monlog to self
  894. -126> 2016-01-14 15:50:21.362105 7f960d8f9700 10 log_client log_queue is 16 last_log 16 sent 15 num 16 unsent 1 sending 1
  895. -125> 2016-01-14 15:50:21.362106 7f960d8f9700 10 log_client will send 2016-01-14 15:50:21.362104 mon.1 10.5.68.230:6789/0 16 : cluster [INF] pgmap v6715571: 4192 pgs: 4 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 4185 active+clean; 31748 GB data, 96182 GB used, 273 TB / 367 TB avail
  896. -124> 2016-01-14 15:50:21.362113 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961ac9d440 con 0x7f961ac818c0
  897. -123> 2016-01-14 15:50:21.362159 7f960d8f9700 0 log_channel(cluster) log [INF] : mdsmap e7932: 1/2/1 up {0=lts-osd1=up:resolve}, 1 up:standby
  898. -122> 2016-01-14 15:50:21.362164 7f960d8f9700 10 log_client _send_to_monlog to self
  899. -121> 2016-01-14 15:50:21.362165 7f960d8f9700 10 log_client log_queue is 17 last_log 17 sent 16 num 17 unsent 1 sending 1
  900. -120> 2016-01-14 15:50:21.362166 7f960d8f9700 10 log_client will send 2016-01-14 15:50:21.362163 mon.1 10.5.68.230:6789/0 17 : cluster [INF] mdsmap e7932: 1/2/1 up {0=lts-osd1=up:resolve}, 1 up:standby
  901. -119> 2016-01-14 15:50:21.362173 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961b375f80 con 0x7f961ac818c0
  902. -118> 2016-01-14 15:50:21.362270 7f960d8f9700 0 log_channel(cluster) log [INF] : osdmap e10235: 102 osds: 101 up, 101 in
  903. -117> 2016-01-14 15:50:21.362275 7f960d8f9700 10 log_client _send_to_monlog to self
  904. -116> 2016-01-14 15:50:21.362276 7f960d8f9700 10 log_client log_queue is 18 last_log 18 sent 17 num 18 unsent 1 sending 1
  905. -115> 2016-01-14 15:50:21.362278 7f960d8f9700 10 log_client will send 2016-01-14 15:50:21.362275 mon.1 10.5.68.230:6789/0 18 : cluster [INF] osdmap e10235: 102 osds: 101 up, 101 in
  906. -114> 2016-01-14 15:50:21.362290 7f960d8f9700 1 -- 10.5.68.230:6789/0 --> 10.5.68.230:6789/0 -- log(1 entries) v1 -- ?+0 0x7f961b375d40 con 0x7f961ac818c0
  907. -113> 2016-01-14 15:50:21.362352 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362352, event: callback finished, op: log(1 entries)
  908. -112> 2016-01-14 15:50:21.362360 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362360, event: psvc:dispatch, op: log(1 entries)
  909. -111> 2016-01-14 15:50:21.362364 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362365 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  910. -110> 2016-01-14 15:50:21.362372 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362372, event: logm:preprocess_query, op: log(1 entries)
  911. -109> 2016-01-14 15:50:21.362375 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362375, event: logm:preprocess_log, op: log(1 entries)
  912. -108> 2016-01-14 15:50:21.362383 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362383, event: logm:prepare_update, op: log(1 entries)
  913. -107> 2016-01-14 15:50:21.362389 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362389, event: logm:prepare_log, op: log(1 entries)
  914. -106> 2016-01-14 15:50:21.362395 7f960d8f9700 5 -- op tracker -- seq: 2, time: 2016-01-14 15:50:21.362395, event: logm:wait_for_finished_proposal, op: log(1 entries)
  915. -105> 2016-01-14 15:50:21.362400 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362400, event: callback finished, op: log(1 entries)
  916. -104> 2016-01-14 15:50:21.362410 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362410, event: psvc:dispatch, op: log(1 entries)
  917. -103> 2016-01-14 15:50:21.362412 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362413 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  918. -102> 2016-01-14 15:50:21.362416 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362416, event: logm:preprocess_query, op: log(1 entries)
  919. -101> 2016-01-14 15:50:21.362418 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362418, event: logm:preprocess_log, op: log(1 entries)
  920. -100> 2016-01-14 15:50:21.362422 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362422, event: logm:prepare_update, op: log(1 entries)
  921. -99> 2016-01-14 15:50:21.362424 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362424, event: logm:prepare_log, op: log(1 entries)
  922. -98> 2016-01-14 15:50:21.362429 7f960d8f9700 5 -- op tracker -- seq: 3, time: 2016-01-14 15:50:21.362429, event: logm:wait_for_finished_proposal, op: log(1 entries)
  923. -97> 2016-01-14 15:50:21.362432 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362432, event: callback finished, op: log(1 entries)
  924. -96> 2016-01-14 15:50:21.362434 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362434, event: psvc:dispatch, op: log(1 entries)
  925. -95> 2016-01-14 15:50:21.362435 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362436 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  926. -94> 2016-01-14 15:50:21.362438 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362438, event: logm:preprocess_query, op: log(1 entries)
  927. -93> 2016-01-14 15:50:21.362440 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362440, event: logm:preprocess_log, op: log(1 entries)
  928. -92> 2016-01-14 15:50:21.362444 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362444, event: logm:prepare_update, op: log(1 entries)
  929. -91> 2016-01-14 15:50:21.362446 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362446, event: logm:prepare_log, op: log(1 entries)
  930. -90> 2016-01-14 15:50:21.362449 7f960d8f9700 5 -- op tracker -- seq: 4, time: 2016-01-14 15:50:21.362449, event: logm:wait_for_finished_proposal, op: log(1 entries)
  931. -89> 2016-01-14 15:50:21.362452 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362452, event: callback finished, op: log(1 entries)
  932. -88> 2016-01-14 15:50:21.362454 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362454, event: psvc:dispatch, op: log(1 entries)
  933. -87> 2016-01-14 15:50:21.362455 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362456 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  934. -86> 2016-01-14 15:50:21.362458 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362458, event: logm:preprocess_query, op: log(1 entries)
  935. -85> 2016-01-14 15:50:21.362460 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362460, event: logm:preprocess_log, op: log(1 entries)
  936. -84> 2016-01-14 15:50:21.362463 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362463, event: logm:prepare_update, op: log(1 entries)
  937. -83> 2016-01-14 15:50:21.362465 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362465, event: logm:prepare_log, op: log(1 entries)
  938. -82> 2016-01-14 15:50:21.362469 7f960d8f9700 5 -- op tracker -- seq: 6, time: 2016-01-14 15:50:21.362469, event: logm:wait_for_finished_proposal, op: log(1 entries)
  939. -81> 2016-01-14 15:50:21.362471 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362471, event: callback finished, op: log(1 entries)
  940. -80> 2016-01-14 15:50:21.362473 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362473, event: psvc:dispatch, op: log(1 entries)
  941. -79> 2016-01-14 15:50:21.362475 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362475 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  942. -78> 2016-01-14 15:50:21.362478 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362478, event: logm:preprocess_query, op: log(1 entries)
  943. -77> 2016-01-14 15:50:21.362480 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362480, event: logm:preprocess_log, op: log(1 entries)
  944. -76> 2016-01-14 15:50:21.362483 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362483, event: logm:prepare_update, op: log(1 entries)
  945. -75> 2016-01-14 15:50:21.362485 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362485, event: logm:prepare_log, op: log(1 entries)
  946. -74> 2016-01-14 15:50:21.362488 7f960d8f9700 5 -- op tracker -- seq: 7, time: 2016-01-14 15:50:21.362488, event: logm:wait_for_finished_proposal, op: log(1 entries)
  947. -73> 2016-01-14 15:50:21.362490 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362490, event: callback finished, op: log(1 entries)
  948. -72> 2016-01-14 15:50:21.362492 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362492, event: psvc:dispatch, op: log(1 entries)
  949. -71> 2016-01-14 15:50:21.362495 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362495 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  950. -70> 2016-01-14 15:50:21.362498 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362498, event: logm:preprocess_query, op: log(1 entries)
  951. -69> 2016-01-14 15:50:21.362500 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362500, event: logm:preprocess_log, op: log(1 entries)
  952. -68> 2016-01-14 15:50:21.362503 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362503, event: logm:prepare_update, op: log(1 entries)
  953. -67> 2016-01-14 15:50:21.362505 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362505, event: logm:prepare_log, op: log(1 entries)
  954. -66> 2016-01-14 15:50:21.362508 7f960d8f9700 5 -- op tracker -- seq: 8, time: 2016-01-14 15:50:21.362508, event: logm:wait_for_finished_proposal, op: log(1 entries)
  955. -65> 2016-01-14 15:50:21.362510 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362510, event: callback finished, op: log(1 entries)
  956. -64> 2016-01-14 15:50:21.362512 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362512, event: psvc:dispatch, op: log(1 entries)
  957. -63> 2016-01-14 15:50:21.362514 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362514 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  958. -62> 2016-01-14 15:50:21.362517 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362517, event: logm:preprocess_query, op: log(1 entries)
  959. -61> 2016-01-14 15:50:21.362520 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362520, event: logm:preprocess_log, op: log(1 entries)
  960. -60> 2016-01-14 15:50:21.362523 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362523, event: logm:prepare_update, op: log(1 entries)
  961. -59> 2016-01-14 15:50:21.362525 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362525, event: logm:prepare_log, op: log(1 entries)
  962. -58> 2016-01-14 15:50:21.362530 7f960d8f9700 5 -- op tracker -- seq: 9, time: 2016-01-14 15:50:21.362530, event: logm:wait_for_finished_proposal, op: log(1 entries)
  963. -57> 2016-01-14 15:50:21.362532 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362532, event: callback finished, op: log(1 entries)
  964. -56> 2016-01-14 15:50:21.362534 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362534, event: psvc:dispatch, op: log(1 entries)
  965. -55> 2016-01-14 15:50:21.362536 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362536 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  966. -54> 2016-01-14 15:50:21.362548 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362548, event: logm:preprocess_query, op: log(1 entries)
  967. -53> 2016-01-14 15:50:21.362550 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362550, event: logm:preprocess_log, op: log(1 entries)
  968. -52> 2016-01-14 15:50:21.362554 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362554, event: logm:prepare_update, op: log(1 entries)
  969. -51> 2016-01-14 15:50:21.362556 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362556, event: logm:prepare_log, op: log(1 entries)
  970. -50> 2016-01-14 15:50:21.362560 7f960d8f9700 5 -- op tracker -- seq: 10, time: 2016-01-14 15:50:21.362560, event: logm:wait_for_finished_proposal, op: log(1 entries)
  971. -49> 2016-01-14 15:50:21.362563 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362563, event: callback finished, op: log(1 entries)
  972. -48> 2016-01-14 15:50:21.362565 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362565, event: psvc:dispatch, op: log(1 entries)
  973. -47> 2016-01-14 15:50:21.362567 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362567 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  974. -46> 2016-01-14 15:50:21.362570 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362570, event: logm:preprocess_query, op: log(1 entries)
  975. -45> 2016-01-14 15:50:21.362572 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362572, event: logm:preprocess_log, op: log(1 entries)
  976. -44> 2016-01-14 15:50:21.362575 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362575, event: logm:prepare_update, op: log(1 entries)
  977. -43> 2016-01-14 15:50:21.362577 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362577, event: logm:prepare_log, op: log(1 entries)
  978. -42> 2016-01-14 15:50:21.362581 7f960d8f9700 5 -- op tracker -- seq: 11, time: 2016-01-14 15:50:21.362581, event: logm:wait_for_finished_proposal, op: log(1 entries)
  979. -41> 2016-01-14 15:50:21.362583 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362583, event: callback finished, op: log(1 entries)
  980. -40> 2016-01-14 15:50:21.362585 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362585, event: psvc:dispatch, op: log(1 entries)
  981. -39> 2016-01-14 15:50:21.362587 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362587 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  982. -38> 2016-01-14 15:50:21.362590 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362590, event: logm:preprocess_query, op: log(1 entries)
  983. -37> 2016-01-14 15:50:21.362592 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362591, event: logm:preprocess_log, op: log(1 entries)
  984. -36> 2016-01-14 15:50:21.362595 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362595, event: logm:prepare_update, op: log(1 entries)
  985. -35> 2016-01-14 15:50:21.362597 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362597, event: logm:prepare_log, op: log(1 entries)
  986. -34> 2016-01-14 15:50:21.362601 7f960d8f9700 5 -- op tracker -- seq: 12, time: 2016-01-14 15:50:21.362601, event: logm:wait_for_finished_proposal, op: log(1 entries)
  987. -33> 2016-01-14 15:50:21.362603 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362603, event: callback finished, op: log(1 entries)
  988. -32> 2016-01-14 15:50:21.362605 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362604, event: psvc:dispatch, op: log(1 entries)
  989. -31> 2016-01-14 15:50:21.362606 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362616 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  990. -30> 2016-01-14 15:50:21.362619 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362619, event: logm:preprocess_query, op: log(1 entries)
  991. -29> 2016-01-14 15:50:21.362621 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362621, event: logm:preprocess_log, op: log(1 entries)
  992. -28> 2016-01-14 15:50:21.362625 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362625, event: logm:prepare_update, op: log(1 entries)
  993. -27> 2016-01-14 15:50:21.362627 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362627, event: logm:prepare_log, op: log(1 entries)
  994. -26> 2016-01-14 15:50:21.362631 7f960d8f9700 5 -- op tracker -- seq: 13, time: 2016-01-14 15:50:21.362631, event: logm:wait_for_finished_proposal, op: log(1 entries)
  995. -25> 2016-01-14 15:50:21.362634 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362634, event: callback finished, op: log(1 entries)
  996. -24> 2016-01-14 15:50:21.362636 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362636, event: psvc:dispatch, op: log(1 entries)
  997. -23> 2016-01-14 15:50:21.362638 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362638 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  998. -22> 2016-01-14 15:50:21.362641 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362641, event: logm:preprocess_query, op: log(1 entries)
  999. -21> 2016-01-14 15:50:21.362643 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362643, event: logm:preprocess_log, op: log(1 entries)
  1000. -20> 2016-01-14 15:50:21.362646 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362646, event: logm:prepare_update, op: log(1 entries)
  1001. -19> 2016-01-14 15:50:21.362648 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362648, event: logm:prepare_log, op: log(1 entries)
  1002. -18> 2016-01-14 15:50:21.362652 7f960d8f9700 5 -- op tracker -- seq: 14, time: 2016-01-14 15:50:21.362652, event: logm:wait_for_finished_proposal, op: log(1 entries)
  1003. -17> 2016-01-14 15:50:21.362654 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362654, event: callback finished, op: log(1 entries)
  1004. -16> 2016-01-14 15:50:21.362663 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362663, event: psvc:dispatch, op: log(1 entries)
  1005. -15> 2016-01-14 15:50:21.362665 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362666 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  1006. -14> 2016-01-14 15:50:21.362669 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362669, event: logm:preprocess_query, op: log(1 entries)
  1007. -13> 2016-01-14 15:50:21.362670 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362670, event: logm:preprocess_log, op: log(1 entries)
  1008. -12> 2016-01-14 15:50:21.362674 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362674, event: logm:prepare_update, op: log(1 entries)
  1009. -11> 2016-01-14 15:50:21.362676 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362676, event: logm:prepare_log, op: log(1 entries)
  1010. -10> 2016-01-14 15:50:21.362679 7f960d8f9700 5 -- op tracker -- seq: 15, time: 2016-01-14 15:50:21.362679, event: logm:wait_for_finished_proposal, op: log(1 entries)
  1011. -9> 2016-01-14 15:50:21.362682 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362682, event: callback finished, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1012. -8> 2016-01-14 15:50:21.362684 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362684, event: psvc:dispatch, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1013. -7> 2016-01-14 15:50:21.362686 7f960d8f9700 5 mon.lts-osd2@1(leader).paxos(paxos active c 13454856..13455395) is_readable = 1 - now=2016-01-14 15:50:21.362686 lease_expire=2016-01-14 15:50:26.361641 has v0 lc 13455395
  1014. -6> 2016-01-14 15:50:21.362693 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362693, event: mdsmap:preprocess_query, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1015. -5> 2016-01-14 15:50:21.362698 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362698, event: mdsmap:preprocess_command, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1016. -4> 2016-01-14 15:50:21.362734 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362734, event: mdsmap:prepare_update, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1017. -3> 2016-01-14 15:50:21.362740 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362740, event: mdsmap:prepare_command, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1018. -2> 2016-01-14 15:50:21.362751 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362751, event: mdsmap:management_command, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1019. -1> 2016-01-14 15:50:21.362757 7f960d8f9700 5 -- op tracker -- seq: 17, time: 2016-01-14 15:50:21.362757, event: mdsmap:filesystem_command, op: mon_command({"prefix": "mds setmap", "epoch": 7821} v 0)
  1020. 0> 2016-01-14 15:50:21.365929 7f960d8f9700 -1 *** Caught signal (Aborted) **
  1021. in thread 7f960d8f9700
  1022.  
  1023. ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
  1024. 1: (()+0x68e43a) [0x7f961728643a]
  1025. 2: (()+0x10340) [0x7f9615cde340]
  1026. 3: (gsignal()+0x39) [0x7f961417fcc9]
  1027. 4: (abort()+0x148) [0x7f96141830d8]
  1028. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9614a8a535]
  1029. 6: (()+0x5e6d6) [0x7f9614a886d6]
  1030. 7: (()+0x5e703) [0x7f9614a88703]
  1031. 8: (()+0x5e922) [0x7f9614a88922]
  1032. 9: (ceph::buffer::list::iterator_impl<false>::copy(unsigned int, char*)+0xa5) [0x7f9617062675]
  1033. 10: (MDSMap::decode(ceph::buffer::list::iterator&)+0x29) [0x7f9616fbb239]
  1034. 11: (MDSMonitor::filesystem_command(std::shared_ptr<MonOpRequest>, std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x6dc) [0x7f9616eef14c]
  1035. 12: (MDSMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x8e3) [0x7f9616ef1a73]
  1036. 13: (MDSMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x2e4) [0x7f9616ef2334]
  1037. 14: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xd7b) [0x7f9616e729db]
  1038. 15: (PaxosService::C_RetryMessage::_finish(int)+0x54) [0x7f9616e75054]
  1039. 16: (C_MonOp::finish(int)+0x69) [0x7f9616e42199]
  1040. 17: (Context::complete(int)+0x9) [0x7f9616e412a9]
  1041. 18: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7f9616e47364]
  1042. 19: (Paxos::finish_round()+0x10b) [0x7f9616e671eb]
  1043. 20: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xea2) [0x7f9616e6b632]
  1044. 21: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x2d4) [0x7f9616e6bf24]
  1045. 22: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xcfb) [0x7f9616e3c48b]
  1046. 23: (Monitor::_ms_dispatch(Message*)+0x628) [0x7f9616e3cd78]
  1047. 24: (Monitor::ms_dispatch(Message*)+0x23) [0x7f9616e5b6a3]
  1048. 25: (DispatchQueue::entry()+0x63a) [0x7f96171d72aa]
  1049. 26: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f961703f06d]
  1050. 27: (()+0x8182) [0x7f9615cd6182]
  1051. 28: (clone()+0x6d) [0x7f961424347d]
  1052. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1053.  
  1054. --- logging levels ---
  1055. 0/ 5 none
  1056. 0/ 1 lockdep
  1057. 0/ 1 context
  1058. 1/ 1 crush
  1059. 1/ 5 mds
  1060. 1/ 5 mds_balancer
  1061. 1/ 5 mds_locker
  1062. 1/ 5 mds_log
  1063. 1/ 5 mds_log_expire
  1064. 1/ 5 mds_migrator
  1065. 0/ 1 buffer
  1066. 0/ 1 timer
  1067. 0/ 1 filer
  1068. 0/ 1 striper
  1069. 0/ 1 objecter
  1070. 0/ 5 rados
  1071. 0/ 5 rbd
  1072. 0/ 5 rbd_replay
  1073. 0/ 5 journaler
  1074. 0/ 5 objectcacher
  1075. 0/ 5 client
  1076. 0/ 5 osd
  1077. 0/ 5 optracker
  1078. 0/ 5 objclass
  1079. 1/ 3 filestore
  1080. 1/ 3 keyvaluestore
  1081. 1/ 3 journal
  1082. 0/ 5 ms
  1083. 1/ 5 mon
  1084. 0/10 monc
  1085. 1/ 5 paxos
  1086. 0/ 5 tp
  1087. 1/ 5 auth
  1088. 1/ 5 crypto
  1089. 1/ 1 finisher
  1090. 1/ 5 heartbeatmap
  1091. 1/ 5 perfcounter
  1092. 1/ 5 rgw
  1093. 1/10 civetweb
  1094. 1/ 5 javaclient
  1095. 1/ 5 asok
  1096. 1/ 1 throttle
  1097. 0/ 0 refs
  1098. 1/ 5 xio
  1099. 1/ 5 compressor
  1100. 1/ 5 newstore
  1101. -2/-2 (syslog threshold)
  1102. -1/-1 (stderr threshold)
  1103. max_recent 10000
  1104. max_new 1000
  1105. log_file /var/log/ceph/ceph-mon.lts-osd2.log
  1106. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!