Guest User

Untitled

a guest
Jul 9th, 2016
51
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. -8> 2016-07-10 00:53:45.503560 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503560, event: callback finished, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  2. -7> 2016-07-10 00:53:45.503585 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503585, event: psvc:dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  3. -6> 2016-07-10 00:53:45.503607 7f0c31c6b700 5 mon.cephmon1@0(leader).paxos(paxos active c 3914095..3914636) is_readable = 1 - now=2016-07-10 00:53:45.503610 lease_expire=2016-07-10 00:53:50.499732 has v0 lc 3914636
  4. -5> 2016-07-10 00:53:45.503637 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503636, event: osdmap:preprocess_query, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  5. -4> 2016-07-10 00:53:45.503661 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503661, event: osdmap:preprocess_command, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  6. -3> 2016-07-10 00:53:45.503776 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503775, event: osdmap:prepare_update, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  7. -2> 2016-07-10 00:53:45.503816 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503815, event: osdmap:prepare_command, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  8. -1> 2016-07-10 00:53:45.503916 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503915, event: osdmap:prepare_command_impl, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  9. 0> 2016-07-10 00:53:45.511588 7f0c31c6b700 -1 *** Caught signal (Segmentation fault) **
  10. in thread 7f0c31c6b700 thread_name:ms_dispatch
  11.  
  12. ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
  13. 1: (()+0x5055ea) [0x564f430df5ea]
  14. 2: (()+0xf100) [0x7f0c391ed100]
  15. 3: (OSDMonitor::prepare_command_pool_set(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x122f) [0x564f42eeb97f]
  16. 4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&)+0xf02c) [0x564f42efb68c]
  17. 5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x64f) [0x564f42f0046f]
  18. 6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x307) [0x564f42f01fc7]
  19. 7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xe0b) [0x564f42eb060b]
  20. 8: (PaxosService::C_RetryMessage::_finish(int)+0x74) [0x564f42eb3db4]
  21. 9: (C_MonOp::finish(int)+0x79) [0x564f42e7e5b9]
  22. 10: (Context::complete(int)+0x9) [0x564f42e7d5f9]
  23. 11: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xac) [0x564f42e83e4c]
  24. 12: (Paxos::finish_round()+0x11e) [0x564f42ea802e]
  25. 13: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xef2) [0x564f42ea94d2]
  26. 14: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x304) [0x564f42ea9ec4]
  27. 15: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xb8a) [0x564f42e789ca]
  28. 16: (Monitor::_ms_dispatch(Message*)+0x6c9) [0x564f42e79519]
  29. 17: (Monitor::ms_dispatch(Message*)+0x23) [0x564f42e9a063]
  30. 18: (DispatchQueue::entry()+0x78a) [0x564f432f2d1a]
  31. 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x564f431e820d]
  32. 20: (()+0x7dc5) [0x7f0c391e5dc5]
  33. 21: (clone()+0x6d) [0x7f0c37aadced]
  34. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  35.  
  36. --- logging levels ---
  37. 0/ 5 none
  38. 0/ 1 lockdep
  39. 0/ 1 context
  40. 1/ 1 crush
  41. 1/ 5 mds
  42. 1/ 5 mds_balancer
  43. 1/ 5 mds_locker
  44. 1/ 5 mds_log
  45. 1/ 5 mds_log_expire
  46. 1/ 5 mds_migrator
  47. 0/ 1 buffer
  48. 0/ 1 timer
  49. 0/ 1 filer
  50. 0/ 1 striper
  51. 0/ 1 objecter
  52. 0/ 5 rados
  53. 0/ 5 rbd
  54. 0/ 5 rbd_mirror
  55. 0/ 5 rbd_replay
  56. 0/ 5 journaler
  57. 0/ 5 objectcacher
  58. 0/ 5 client
  59. 0/ 5 osd
  60. 0/ 5 optracker
  61. 0/ 5 objclass
  62. 1/ 3 filestore
  63. 1/ 3 journal
  64. 0/ 5 ms
  65. 1/ 5 mon
  66. 0/10 monc
  67. 1/ 5 paxos
  68. 0/ 5 tp
  69. 1/ 5 auth
  70. 1/ 5 crypto
  71. 1/ 1 finisher
  72. 1/ 5 heartbeatmap
  73. 1/ 5 perfcounter
  74. 1/ 5 rgw
  75. 1/10 civetweb
  76. 1/ 5 javaclient
  77. 1/ 5 asok
  78. 1/ 1 throttle
  79. 0/ 0 refs
  80. 1/ 5 xio
  81. 1/ 5 compressor
  82. 1/ 5 newstore
  83. 1/ 5 bluestore
  84. 1/ 5 bluefs
  85. 1/ 3 bdev
  86. 1/ 5 kstore
  87. 4/ 5 rocksdb
  88. 4/ 5 leveldb
  89. 1/ 5 kinetic
  90. 1/ 5 fuse
  91. -2/-2 (syslog threshold)
  92. -1/-1 (stderr threshold)
  93. max_recent 10000
  94. max_new 1000
  95. log_file /var/log/ceph/ceph-mon.cephmon1.log
  96. --- end dump of recent events ---
  97. 2016-07-10 00:53:45.876504 7fdd97da64c0 0 set uid:gid to 167:167 (ceph:ceph)
  98. 2016-07-10 00:53:45.876574 7fdd97da64c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mon, pid 17937
  99. 2016-07-10 00:53:45.876683 7fdd97da64c0 0 pidfile_write: ignore empty --pid-file
  100. 2016-07-10 00:53:46.021225 7fdd97da64c0 1 leveldb: Recovering log #525160
  101. 2016-07-10 00:53:46.021398 7fdd97da64c0 1 leveldb: Level-0 table #525162: started
  102. 2016-07-10 00:53:46.023255 7fdd97da64c0 1 leveldb: Level-0 table #525162: 192 bytes OK
  103. 2016-07-10 00:53:46.025897 7fdd97da64c0 1 leveldb: Delete type=0 #525160
  104.  
  105. 2016-07-10 00:53:46.026171 7fdd97da64c0 1 leveldb: Delete type=3 #525158
  106.  
  107. 2016-07-10 00:53:46.027545 7fdd97da64c0 0 starting mon.cephmon1 rank 0 at 10.0.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon1 fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  108. 2016-07-10 00:53:46.028204 7fdd97da64c0 1 mon.cephmon1@-1(probing) e1 preinit fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  109. 2016-07-10 00:53:46.029412 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(pgmap 1943096..1943707) refresh upgraded, format 0 -> 1
  110. 2016-07-10 00:53:46.029449 7fdd97da64c0 1 mon.cephmon1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
  111. 2016-07-10 00:53:46.085370 7fdd97da64c0 0 mon.cephmon1@-1(probing).mds e46 print_map
  112. e46
  113. enable_multiple, ever_enabled_multiple: 0,0
  114. compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  115.  
  116. Filesystem 'ceph-gen2' (2)
  117. fs_name ceph-gen2
  118. epoch 43
  119. flags 0
  120. created 2016-06-11 21:53:02.142649
  121. modified 2016-06-14 11:09:16.783356
  122. tableserver 0
  123. root 0
  124. session_timeout 60
  125. session_autoclose 300
  126. max_file_size 1099511627776
  127. last_failure 0
  128. last_failure_osd_epoch 216
  129. compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  130. max_mds 1
  131. in 0
  132. up {0=54100}
  133. failed
  134. damaged
  135. stopped
  136. data_pools 4
  137. metadata_pool 5
  138. inline_data disabled
  139. 54100: 10.0.0.12:6800/12411 'cephmon2' mds.0.39 up:active seq 7
  140.  
  141.  
  142. Standby daemons:
  143.  
  144. 44097: 10.0.0.11:6800/5378 'cephmon1' mds.-1.0 up:standby seq 539373
  145. 54103: 10.0.0.13:6800/12460 'cephmon3' mds.-1.0 up:standby seq 539371
  146.  
  147. 2016-07-10 00:53:46.086399 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  148. 2016-07-10 00:53:46.086423 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  149. 2016-07-10 00:53:46.086438 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  150. 2016-07-10 00:53:46.086448 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  151. 2016-07-10 00:53:46.087714 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(auth 751..888) refresh upgraded, format 0 -> 1
  152. 2016-07-10 00:53:46.092721 7fdd97da64c0 0 mon.cephmon1@-1(probing) e1 my rank is now 0 (was -1)
  153. 2016-07-10 00:53:46.095724 7fdd8d3bd700 0 -- 10.0.0.11:6789/0 >> 10.0.0.12:6789/0 pipe(0x5587f8676000 sd=11 :37630 s=2 pgs=2493 cs=1 l=0 c=0x5587f8275600).reader missed message? skipped from seq 0 to 699733957
  154. 2016-07-10 00:53:46.096136 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1 calling new monitor election
  155. 2016-07-10 00:53:46.096868 7fdd8f5c3700 1 mon.cephmon1@0(electing).elector(90) init, last seen epoch 90
  156. 2016-07-10 00:53:46.096850 7fdd8d2bc700 0 -- 10.0.0.11:6789/0 >> 10.0.0.13:6789/0 pipe(0x5587f8677400 sd=12 :43136 s=2 pgs=3635 cs=1 l=0 c=0x5587f8275780).reader missed message? skipped from seq 0 to 233744893
  157. 2016-07-10 00:53:46.107284 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1@0 won leader election with quorum 0,1,2
  158. 2016-07-10 00:53:46.109361 7fdd8f5c3700 0 log_channel(cluster) log [INF] : HEALTH_OK
  159. 2016-07-10 00:53:46.187279 7fdd8f5c3700 0 mon.cephmon1@0(leader) e1 handle_command mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1
  160. 2016-07-10 00:53:46.187410 7fdd8f5c3700 0 log_channel(audit) log [INF] : from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  161. 2016-07-10 00:53:46.198494 7fdd8f5c3700 0 log_channel(cluster) log [INF] : monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  162. 2016-07-10 00:53:46.198644 7fdd8f5c3700 0 log_channel(cluster) log [INF] : pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  163. 2016-07-10 00:53:46.198847 7fdd8f5c3700 0 log_channel(cluster) log [INF] : fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  164. 2016-07-10 00:53:46.199056 7fdd8f5c3700 0 log_channel(cluster) log [INF] : osdmap e474: 16 osds: 16 up, 16 in
  165. 2016-07-10 00:53:46.208221 7fdd8f5c3700 -1 *** Caught signal (Segmentation fault) **
  166. in thread 7fdd8f5c3700 thread_name:ms_dispatch
  167.  
  168. ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
  169. 1: (()+0x5055ea) [0x5587ec4995ea]
  170. 2: (()+0xf100) [0x7fdd96b45100]
  171. 3: (OSDMonitor::prepare_command_pool_set(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x122f) [0x5587ec2a597f]
  172. 4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&)+0xf02c) [0x5587ec2b568c]
  173. 5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x64f) [0x5587ec2ba46f]
  174. 6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x307) [0x5587ec2bbfc7]
  175. 7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xe0b) [0x5587ec26a60b]
  176. 8: (PaxosService::C_RetryMessage::_finish(int)+0x74) [0x5587ec26ddb4]
  177. 9: (C_MonOp::finish(int)+0x79) [0x5587ec2385b9]
  178. 10: (Context::complete(int)+0x9) [0x5587ec2375f9]
  179. 11: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xac) [0x5587ec23de4c]
  180. 12: (Paxos::finish_round()+0x11e) [0x5587ec26202e]
  181. 13: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xef2) [0x5587ec2634d2]
  182. 14: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x304) [0x5587ec263ec4]
  183. 15: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xb8a) [0x5587ec2329ca]
  184. 16: (Monitor::_ms_dispatch(Message*)+0x6c9) [0x5587ec233519]
  185. 17: (Monitor::ms_dispatch(Message*)+0x23) [0x5587ec254063]
  186. 18: (DispatchQueue::entry()+0x78a) [0x5587ec6acd1a]
  187. 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x5587ec5a220d]
  188. 20: (()+0x7dc5) [0x7fdd96b3ddc5]
  189. 21: (clone()+0x6d) [0x7fdd95405ced]
  190. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  191.  
  192. --- begin dump of recent events ---
  193. -355> 2016-07-10 00:53:45.851883 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perfcounters_dump hook 0x5587f81ca030
  194. -354> 2016-07-10 00:53:45.851939 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 1 hook 0x5587f81ca030
  195. -353> 2016-07-10 00:53:45.851960 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf dump hook 0x5587f81ca030
  196. -352> 2016-07-10 00:53:45.851979 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perfcounters_schema hook 0x5587f81ca030
  197. -351> 2016-07-10 00:53:45.851996 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 2 hook 0x5587f81ca030
  198. -350> 2016-07-10 00:53:45.852013 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf schema hook 0x5587f81ca030
  199. -349> 2016-07-10 00:53:45.852053 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf reset hook 0x5587f81ca030
  200. -348> 2016-07-10 00:53:45.852065 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config show hook 0x5587f81ca030
  201. -347> 2016-07-10 00:53:45.852077 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config set hook 0x5587f81ca030
  202. -346> 2016-07-10 00:53:45.852097 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config get hook 0x5587f81ca030
  203. -345> 2016-07-10 00:53:45.852117 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config diff hook 0x5587f81ca030
  204. -344> 2016-07-10 00:53:45.852408 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log flush hook 0x5587f81ca030
  205. -343> 2016-07-10 00:53:45.852431 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log dump hook 0x5587f81ca030
  206. -342> 2016-07-10 00:53:45.852443 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log reopen hook 0x5587f81ca030
  207. -341> 2016-07-10 00:53:45.876504 7fdd97da64c0 0 set uid:gid to 167:167 (ceph:ceph)
  208. -340> 2016-07-10 00:53:45.876574 7fdd97da64c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mon, pid 17937
  209. -339> 2016-07-10 00:53:45.876683 7fdd97da64c0 0 pidfile_write: ignore empty --pid-file
  210. -338> 2016-07-10 00:53:45.993655 7fdd97da64c0 5 asok(0x5587f82a2000) init /var/run/ceph/ceph-mon.cephmon1.asok
  211. -337> 2016-07-10 00:53:45.993689 7fdd97da64c0 5 asok(0x5587f82a2000) bind_and_listen /var/run/ceph/ceph-mon.cephmon1.asok
  212. -336> 2016-07-10 00:53:45.993991 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 0 hook 0x5587f81c6090
  213. -335> 2016-07-10 00:53:45.994016 7fdd97da64c0 5 asok(0x5587f82a2000) register_command version hook 0x5587f81c6090
  214. -334> 2016-07-10 00:53:45.994058 7fdd97da64c0 5 asok(0x5587f82a2000) register_command git_version hook 0x5587f81c6090
  215. -333> 2016-07-10 00:53:45.994080 7fdd97da64c0 5 asok(0x5587f82a2000) register_command help hook 0x5587f81ca130
  216. -332> 2016-07-10 00:53:45.994093 7fdd97da64c0 5 asok(0x5587f82a2000) register_command get_command_descriptions hook 0x5587f81ca140
  217. -331> 2016-07-10 00:53:45.994330 7fdd92f27700 5 asok(0x5587f82a2000) entry start
  218. -330> 2016-07-10 00:53:46.021225 7fdd97da64c0 1 leveldb: Recovering log #525160
  219. -329> 2016-07-10 00:53:46.021398 7fdd97da64c0 1 leveldb: Level-0 table #525162: started
  220. -328> 2016-07-10 00:53:46.023255 7fdd97da64c0 1 leveldb: Level-0 table #525162: 192 bytes OK
  221. -327> 2016-07-10 00:53:46.025897 7fdd97da64c0 1 leveldb: Delete type=0 #525160
  222.  
  223. -326> 2016-07-10 00:53:46.026171 7fdd97da64c0 1 leveldb: Delete type=3 #525158
  224.  
  225. -325> 2016-07-10 00:53:46.027545 7fdd97da64c0 0 starting mon.cephmon1 rank 0 at 10.0.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon1 fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  226. -324> 2016-07-10 00:53:46.027648 7fdd97da64c0 1 -- 10.0.0.11:6789/0 learned my addr 10.0.0.11:6789/0
  227. -323> 2016-07-10 00:53:46.027668 7fdd97da64c0 1 accepter.accepter.bind my_inst.addr is 10.0.0.11:6789/0 need_addr=0
  228. -322> 2016-07-10 00:53:46.027805 7fdd97da64c0 5 adding auth protocol: cephx
  229. -321> 2016-07-10 00:53:46.027825 7fdd97da64c0 5 adding auth protocol: cephx
  230. -320> 2016-07-10 00:53:46.027886 7fdd97da64c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  231. -319> 2016-07-10 00:53:46.027914 7fdd97da64c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  232. -318> 2016-07-10 00:53:46.028204 7fdd97da64c0 1 mon.cephmon1@-1(probing) e1 preinit fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  233. -317> 2016-07-10 00:53:46.029412 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(pgmap 1943096..1943707) refresh upgraded, format 0 -> 1
  234. -316> 2016-07-10 00:53:46.029449 7fdd97da64c0 1 mon.cephmon1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
  235. -315> 2016-07-10 00:53:46.085345 7fdd97da64c0 4 mon.cephmon1@-1(probing).mds e46 new map
  236. -314> 2016-07-10 00:53:46.085370 7fdd97da64c0 0 mon.cephmon1@-1(probing).mds e46 print_map
  237. e46
  238. enable_multiple, ever_enabled_multiple: 0,0
  239. compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  240.  
  241. Filesystem 'ceph-gen2' (2)
  242. fs_name ceph-gen2
  243. epoch 43
  244. flags 0
  245. created 2016-06-11 21:53:02.142649
  246. modified 2016-06-14 11:09:16.783356
  247. tableserver 0
  248. root 0
  249. session_timeout 60
  250. session_autoclose 300
  251. max_file_size 1099511627776
  252. last_failure 0
  253. last_failure_osd_epoch 216
  254. compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  255. max_mds 1
  256. in 0
  257. up {0=54100}
  258. failed
  259. damaged
  260. stopped
  261. data_pools 4
  262. metadata_pool 5
  263. inline_data disabled
  264. 54100: 10.0.0.12:6800/12411 'cephmon2' mds.0.39 up:active seq 7
  265.  
  266.  
  267. Standby daemons:
  268.  
  269. 44097: 10.0.0.11:6800/5378 'cephmon1' mds.-1.0 up:standby seq 539373
  270. 54103: 10.0.0.13:6800/12460 'cephmon3' mds.-1.0 up:standby seq 539371
  271.  
  272. -313> 2016-07-10 00:53:46.086399 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  273. -312> 2016-07-10 00:53:46.086423 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  274. -311> 2016-07-10 00:53:46.086438 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  275. -310> 2016-07-10 00:53:46.086448 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  276. -309> 2016-07-10 00:53:46.087714 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(auth 751..888) refresh upgraded, format 0 -> 1
  277. -308> 2016-07-10 00:53:46.092072 7fdd97da64c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cephmon1/keyring
  278. -307> 2016-07-10 00:53:46.092096 7fdd97da64c0 5 asok(0x5587f82a2000) register_command mon_status hook 0x5587f81ca5f0
  279. -306> 2016-07-10 00:53:46.092112 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum_status hook 0x5587f81ca5f0
  280. -305> 2016-07-10 00:53:46.092124 7fdd97da64c0 5 asok(0x5587f82a2000) register_command sync_force hook 0x5587f81ca5f0
  281. -304> 2016-07-10 00:53:46.092143 7fdd97da64c0 5 asok(0x5587f82a2000) register_command add_bootstrap_peer_hint hook 0x5587f81ca5f0
  282. -303> 2016-07-10 00:53:46.092162 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum enter hook 0x5587f81ca5f0
  283. -302> 2016-07-10 00:53:46.092180 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum exit hook 0x5587f81ca5f0
  284. -301> 2016-07-10 00:53:46.092199 7fdd97da64c0 5 asok(0x5587f82a2000) register_command ops hook 0x5587f81ca5f0
  285. -300> 2016-07-10 00:53:46.092270 7fdd97da64c0 1 -- 10.0.0.11:6789/0 messenger.start
  286. -299> 2016-07-10 00:53:46.092408 7fdd97da64c0 2 mon.cephmon1@-1(probing) e1 init
  287. -298> 2016-07-10 00:53:46.092643 7fdd97da64c0 1 accepter.accepter.start
  288. -297> 2016-07-10 00:53:46.092721 7fdd97da64c0 0 mon.cephmon1@-1(probing) e1 my rank is now 0 (was -1)
  289. -296> 2016-07-10 00:53:46.092739 7fdd97da64c0 1 -- 10.0.0.11:6789/0 mark_down_all
  290. -295> 2016-07-10 00:53:46.092771 7fdd97da64c0 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- mon_probe(probe a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon1) v6 -- ?+0 0x5587f861ed00
  291. -294> 2016-07-10 00:53:46.092941 7fdd97da64c0 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- mon_probe(probe a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon1) v6 -- ?+0 0x5587f861ef80
  292. -293> 2016-07-10 00:53:46.095724 7fdd8d3bd700 0 -- 10.0.0.11:6789/0 >> 10.0.0.12:6789/0 pipe(0x5587f8676000 sd=11 :37630 s=2 pgs=2493 cs=1 l=0 c=0x5587f8275600).reader missed message? skipped from seq 0 to 699733957
  293. -292> 2016-07-10 00:53:46.095876 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733957 ==== mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 )) v6 ==== 567+0+0 (3214913024 0 0) 0x5587f861ed00 con 0x5587f8275600
  294. -291> 2016-07-10 00:53:46.095951 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.095950, event: mon:_ms_dispatch, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  295. -290> 2016-07-10 00:53:46.095990 7fdd8f5c3700 5 mon.cephmon1@0(probing) e1 _ms_dispatch setting monitor caps on this connection
  296. -289> 2016-07-10 00:53:46.096016 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.096015, event: mon:dispatch_op, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  297. -288> 2016-07-10 00:53:46.096136 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1 calling new monitor election
  298. -287> 2016-07-10 00:53:46.096154 7fdd8f5c3700 10 log_client _send_to_monlog to self
  299. -286> 2016-07-10 00:53:46.096160 7fdd8f5c3700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
  300. -285> 2016-07-10 00:53:46.096167 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.096151 mon.0 10.0.0.11:6789/0 1 : cluster [INF] mon.cephmon1 calling new monitor election
  301. -284> 2016-07-10 00:53:46.096204 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 1 at 2016-07-10 00:53:46.096151) v1 -- ?+0 0x5587f829f680 con 0x5587f8274e80
  302. -283> 2016-07-10 00:53:46.096284 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(0) start -- can i be leader?
  303. -282> 2016-07-10 00:53:46.096868 7fdd8f5c3700 1 mon.cephmon1@0(electing).elector(90) init, last seen epoch 90
  304. -281> 2016-07-10 00:53:46.096850 7fdd8d2bc700 0 -- 10.0.0.11:6789/0 >> 10.0.0.13:6789/0 pipe(0x5587f8677400 sd=12 :43136 s=2 pgs=3635 cs=1 l=0 c=0x5587f8275780).reader missed message? skipped from seq 0 to 233744893
  305. -280> 2016-07-10 00:53:46.100941 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 propose 91) v5 -- ?+0 0x5587f8327700
  306. -279> 2016-07-10 00:53:46.101063 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 propose 91) v5 -- ?+0 0x5587f83279c0
  307. -278> 2016-07-10 00:53:46.101122 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.101120, event: done, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  308. -277> 2016-07-10 00:53:46.101186 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 1 at 2016-07-10 00:53:46.096151) v1 ==== 0+0+0 (0 0 0) 0x5587f829f680 con 0x5587f8274e80
  309. -276> 2016-07-10 00:53:46.101238 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101238, event: mon:_ms_dispatch, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  310. -275> 2016-07-10 00:53:46.101261 7fdd8f5c3700 5 mon.cephmon1@0(electing) e1 _ms_dispatch setting monitor caps on this connection
  311. -274> 2016-07-10 00:53:46.101276 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101276, event: mon:dispatch_op, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  312. -273> 2016-07-10 00:53:46.101304 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101303, event: psvc:dispatch, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  313. -272> 2016-07-10 00:53:46.101323 7fdd8f5c3700 5 mon.cephmon1@0(electing).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.101326 lease_expire=0.000000 has v0 lc 3914636
  314. -271> 2016-07-10 00:53:46.101355 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101355, event: logm:wait_for_readable, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  315. -270> 2016-07-10 00:53:46.101382 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101382, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  316. -269> 2016-07-10 00:53:46.101407 7fdd8f5c3700 5 mon.cephmon1@0(electing).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.101410 lease_expire=0.000000 has v0 lc 3914636
  317. -268> 2016-07-10 00:53:46.101437 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101435, event: paxos:wait_for_readable, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  318. -267> 2016-07-10 00:53:46.101470 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744893 ==== mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 )) v6 ==== 567+0+0 (2951463228 0 0) 0x5587f861ef80 con 0x5587f8275780
  319. -266> 2016-07-10 00:53:46.101543 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101542, event: mon:_ms_dispatch, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  320. -265> 2016-07-10 00:53:46.101575 7fdd8f5c3700 5 mon.cephmon1@0(electing) e1 _ms_dispatch setting monitor caps on this connection
  321. -264> 2016-07-10 00:53:46.101596 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101596, event: mon:dispatch_op, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  322. -263> 2016-07-10 00:53:46.101642 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101641, event: done, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  323. -262> 2016-07-10 00:53:46.106174 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744894 ==== election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91) v5 ==== 31040+0+0 (2406017146 0 0) 0x5587f83279c0 con 0x5587f8275780
  324. -261> 2016-07-10 00:53:46.106236 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106233, event: mon:_ms_dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  325. -260> 2016-07-10 00:53:46.106266 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106266, event: mon:dispatch_op, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  326. -259> 2016-07-10 00:53:46.106294 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106293, event: elector:dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  327. -258> 2016-07-10 00:53:46.106345 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106344, event: elector:handle_ack, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  328. -257> 2016-07-10 00:53:46.106368 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) handle_ack from mon.2
  329. -256> 2016-07-10 00:53:46.106381 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) so far i have {0=576460752032874495,2=576460752032874495}
  330. -255> 2016-07-10 00:53:46.106401 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106400, event: done, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  331. -254> 2016-07-10 00:53:46.106487 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733958 ==== election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91) v5 ==== 31040+0+0 (2406017146 0 0) 0x5587f8327700 con 0x5587f8275600
  332. -253> 2016-07-10 00:53:46.106532 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106531, event: mon:_ms_dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  333. -252> 2016-07-10 00:53:46.106557 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106557, event: mon:dispatch_op, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  334. -251> 2016-07-10 00:53:46.106582 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106582, event: elector:dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  335. -250> 2016-07-10 00:53:46.106623 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106622, event: elector:handle_ack, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  336. -249> 2016-07-10 00:53:46.106644 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) handle_ack from mon.1
  337. -248> 2016-07-10 00:53:46.106654 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) so far i have {0=576460752032874495,1=576460752032874495,2=576460752032874495}
  338. -247> 2016-07-10 00:53:46.107157 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 victory 92) v5 -- ?+0 0x5587f8327c80
  339. -246> 2016-07-10 00:53:46.107219 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 victory 92) v5 -- ?+0 0x5587f8327180
  340. -245> 2016-07-10 00:53:46.107284 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1@0 won leader election with quorum 0,1,2
  341. -244> 2016-07-10 00:53:46.107306 7fdd8f5c3700 10 log_client _send_to_monlog to self
  342. -243> 2016-07-10 00:53:46.107311 7fdd8f5c3700 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
  343. -242> 2016-07-10 00:53:46.107319 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.107300 mon.0 10.0.0.11:6789/0 2 : cluster [INF] mon.cephmon1@0 won leader election with quorum 0,1,2
  344. -241> 2016-07-10 00:53:46.107346 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 2 at 2016-07-10 00:53:46.107300) v1 -- ?+0 0x5587f829fb00 con 0x5587f8274e80
  345. -240> 2016-07-10 00:53:46.107901 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- paxos(collect lc 3914636 fc 3914095 pn 4600 opn 0) v3 -- ?+0 0x5587f861ed00
  346. -239> 2016-07-10 00:53:46.107952 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- paxos(collect lc 3914636 fc 3914095 pn 4600 opn 0) v3 -- ?+0 0x5587f861ef80
  347. -238> 2016-07-10 00:53:46.108083 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- time_check( ping e 92 r 1 ) v1 -- ?+0 0x5587f829f440
  348. -237> 2016-07-10 00:53:46.108134 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- time_check( ping e 92 r 1 ) v1 -- ?+0 0x5587f829fd40
  349. -236> 2016-07-10 00:53:46.109361 7fdd8f5c3700 0 log_channel(cluster) log [INF] : HEALTH_OK
  350. -235> 2016-07-10 00:53:46.109383 7fdd8f5c3700 10 log_client _send_to_monlog to self
  351. -234> 2016-07-10 00:53:46.109388 7fdd8f5c3700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
  352. -233> 2016-07-10 00:53:46.109396 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.109379 mon.0 10.0.0.11:6789/0 3 : cluster [INF] HEALTH_OK
  353. -232> 2016-07-10 00:53:46.109442 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 3 at 2016-07-10 00:53:46.109379) v1 -- ?+0 0x5587f82a0400 con 0x5587f8274e80
  354. -231> 2016-07-10 00:53:46.184528 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.184526, event: done, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  355. -230> 2016-07-10 00:53:46.184622 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 2 at 2016-07-10 00:53:46.107300) v1 ==== 0+0+0 (0 0 0) 0x5587f829fb00 con 0x5587f8274e80
  356. -229> 2016-07-10 00:53:46.184692 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184691, event: mon:_ms_dispatch, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  357. -228> 2016-07-10 00:53:46.184715 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184713, event: mon:dispatch_op, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  358. -227> 2016-07-10 00:53:46.184740 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184740, event: psvc:dispatch, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  359. -226> 2016-07-10 00:53:46.184765 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.184768 lease_expire=0.000000 has v0 lc 3914636
  360. -225> 2016-07-10 00:53:46.184797 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184797, event: logm:wait_for_readable, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  361. -224> 2016-07-10 00:53:46.184823 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184822, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  362. -223> 2016-07-10 00:53:46.184846 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.184849 lease_expire=0.000000 has v0 lc 3914636
  363. -222> 2016-07-10 00:53:46.184865 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184864, event: paxos:wait_for_readable, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  364. -221> 2016-07-10 00:53:46.184892 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733961 ==== forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader v3 ==== 638+0+0 (2780614426 0 0) 0x5587f820fe00 con 0x5587f8275600
  365. -220> 2016-07-10 00:53:46.184942 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.184942, event: mon:_ms_dispatch, op: forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader
  366. -219> 2016-07-10 00:53:46.184973 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.184972, event: mon:dispatch_op, op: forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader
  367. -218> 2016-07-10 00:53:46.185090 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185090, event: mon:_ms_dispatch, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  368. -217> 2016-07-10 00:53:46.185121 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185121, event: mon:dispatch_op, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  369. -216> 2016-07-10 00:53:46.185148 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185146, event: psvc:dispatch, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  370. -215> 2016-07-10 00:53:46.185170 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185177 lease_expire=0.000000 has v0 lc 3914636
  371. -214> 2016-07-10 00:53:46.185203 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185201, event: logm:wait_for_readable, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  372. -213> 2016-07-10 00:53:46.185227 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185227, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  373. -212> 2016-07-10 00:53:46.185251 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185254 lease_expire=0.000000 has v0 lc 3914636
  374. -211> 2016-07-10 00:53:46.185276 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185275, event: paxos:wait_for_readable, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  375. -210> 2016-07-10 00:53:46.185301 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.185301, event: done, op: forward(??? ) to leader
  376. -209> 2016-07-10 00:53:46.185342 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 3 at 2016-07-10 00:53:46.109379) v1 ==== 0+0+0 (0 0 0) 0x5587f82a0400 con 0x5587f8274e80
  377. -208> 2016-07-10 00:53:46.185387 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185387, event: mon:_ms_dispatch, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  378. -207> 2016-07-10 00:53:46.185413 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185413, event: mon:dispatch_op, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  379. -206> 2016-07-10 00:53:46.185437 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185437, event: psvc:dispatch, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  380. -205> 2016-07-10 00:53:46.185460 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185463 lease_expire=0.000000 has v0 lc 3914636
  381. -204> 2016-07-10 00:53:46.185486 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185485, event: logm:wait_for_readable, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  382. -203> 2016-07-10 00:53:46.185511 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185509, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  383. -202> 2016-07-10 00:53:46.185533 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185536 lease_expire=0.000000 has v0 lc 3914636
  384. -201> 2016-07-10 00:53:46.185558 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185557, event: paxos:wait_for_readable, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  385. -200> 2016-07-10 00:53:46.185584 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733959 ==== forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader v3 ==== 12320+0+0 (2403987416 0 0) 0x5587f820f500 con 0x5587f8275600
  386. -199> 2016-07-10 00:53:46.185628 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185627, event: mon:_ms_dispatch, op: forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader
  387. -198> 2016-07-10 00:53:46.185684 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185656, event: mon:dispatch_op, op: forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader
  388. -197> 2016-07-10 00:53:46.185733 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185732, event: mon:_ms_dispatch, op: pg_stats(21 pgs tid 131361 v 0)
  389. -196> 2016-07-10 00:53:46.185757 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185756, event: mon:dispatch_op, op: pg_stats(21 pgs tid 131361 v 0)
  390. -195> 2016-07-10 00:53:46.185777 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185777, event: psvc:dispatch, op: pg_stats(21 pgs tid 131361 v 0)
  391. -194> 2016-07-10 00:53:46.185795 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185798 lease_expire=0.000000 has v0 lc 3914636
  392. -193> 2016-07-10 00:53:46.185820 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185820, event: pgmap:wait_for_readable, op: pg_stats(21 pgs tid 131361 v 0)
  393. -192> 2016-07-10 00:53:46.185841 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185840, event: pgmap:wait_for_readable/paxos, op: pg_stats(21 pgs tid 131361 v 0)
  394. -191> 2016-07-10 00:53:46.185858 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185861 lease_expire=0.000000 has v0 lc 3914636
  395. -190> 2016-07-10 00:53:46.185882 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185881, event: paxos:wait_for_readable, op: pg_stats(21 pgs tid 131361 v 0)
  396. -189> 2016-07-10 00:53:46.185903 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185903, event: done, op: forward(??? ) to leader
  397. -188> 2016-07-10 00:53:46.185935 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744895 ==== forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader v3 ==== 2132+0+0 (4169066638 0 0) 0x5587f820f200 con 0x5587f8275780
  398. -187> 2016-07-10 00:53:46.185978 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.185977, event: mon:_ms_dispatch, op: forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader
  399. -186> 2016-07-10 00:53:46.186007 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.186006, event: mon:dispatch_op, op: forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader
  400. -185> 2016-07-10 00:53:46.186063 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186063, event: mon:_ms_dispatch, op: pg_stats(3 pgs tid 131363 v 0)
  401. -184> 2016-07-10 00:53:46.186087 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186086, event: mon:dispatch_op, op: pg_stats(3 pgs tid 131363 v 0)
  402. -183> 2016-07-10 00:53:46.186108 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186107, event: psvc:dispatch, op: pg_stats(3 pgs tid 131363 v 0)
  403. -182> 2016-07-10 00:53:46.186200 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.186204 lease_expire=0.000000 has v0 lc 3914636
  404. -181> 2016-07-10 00:53:46.186234 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186234, event: pgmap:wait_for_readable, op: pg_stats(3 pgs tid 131363 v 0)
  405. -180> 2016-07-10 00:53:46.186256 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186255, event: pgmap:wait_for_readable/paxos, op: pg_stats(3 pgs tid 131363 v 0)
  406. -179> 2016-07-10 00:53:46.186279 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.186282 lease_expire=0.000000 has v0 lc 3914636
  407. -178> 2016-07-10 00:53:46.186306 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186305, event: paxos:wait_for_readable, op: pg_stats(3 pgs tid 131363 v 0)
  408. -177> 2016-07-10 00:53:46.186328 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.186327, event: done, op: forward(??? ) to leader
  409. -176> 2016-07-10 00:53:46.186366 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733963 ==== mon_metadata magic: 0 v1 ==== 414+0+0 (2638027890 0 0) 0x5587f82ea800 con 0x5587f8275600
  410. -175> 2016-07-10 00:53:46.186408 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186407, event: mon:_ms_dispatch, op: mon_metadata magic: 0
  411. -174> 2016-07-10 00:53:46.186431 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186430, event: mon:dispatch_op, op: mon_metadata magic: 0
  412. -173> 2016-07-10 00:53:46.186861 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186860, event: done, op: mon_metadata magic: 0
  413. -172> 2016-07-10 00:53:46.186909 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733960 ==== forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader v3 ==== 383+0+0 (3863550525 0 0) 0x5587f820f800 con 0x5587f8275600
  414. -171> 2016-07-10 00:53:46.186955 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.186954, event: mon:_ms_dispatch, op: forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader
  415. -170> 2016-07-10 00:53:46.186985 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.186983, event: mon:dispatch_op, op: forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader
  416. -169> 2016-07-10 00:53:46.187048 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187048, event: mon:_ms_dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  417. -168> 2016-07-10 00:53:46.187073 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187073, event: mon:dispatch_op, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  418. -167> 2016-07-10 00:53:46.187279 7fdd8f5c3700 0 mon.cephmon1@0(leader) e1 handle_command mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1
  419. -166> 2016-07-10 00:53:46.187410 7fdd8f5c3700 0 log_channel(audit) log [INF] : from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  420. -165> 2016-07-10 00:53:46.187442 7fdd8f5c3700 10 log_client _send_to_monlog to self
  421. -164> 2016-07-10 00:53:46.187449 7fdd8f5c3700 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
  422. -163> 2016-07-10 00:53:46.187477 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.187434 mon.0 10.0.0.11:6789/0 4 : audit [INF] from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  423. -162> 2016-07-10 00:53:46.187516 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 4 at 2016-07-10 00:53:46.187434) v1 -- ?+0 0x5587f82a01c0 con 0x5587f8274e80
  424. -161> 2016-07-10 00:53:46.187566 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187565, event: psvc:dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  425. -160> 2016-07-10 00:53:46.187589 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187592 lease_expire=0.000000 has v0 lc 3914636
  426. -159> 2016-07-10 00:53:46.187618 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187616, event: osdmap:wait_for_readable, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  427. -158> 2016-07-10 00:53:46.187641 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187640, event: osdmap:wait_for_readable/paxos, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  428. -157> 2016-07-10 00:53:46.187660 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187663 lease_expire=0.000000 has v0 lc 3914636
  429. -156> 2016-07-10 00:53:46.187684 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187684, event: paxos:wait_for_readable, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  430. -155> 2016-07-10 00:53:46.187716 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.187715, event: done, op: forward(??? ) to leader
  431. -154> 2016-07-10 00:53:46.187755 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 4 at 2016-07-10 00:53:46.187434) v1 ==== 0+0+0 (0 0 0) 0x5587f82a01c0 con 0x5587f8274e80
  432. -153> 2016-07-10 00:53:46.187800 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187799, event: mon:_ms_dispatch, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  433. -152> 2016-07-10 00:53:46.187826 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187826, event: mon:dispatch_op, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  434. -151> 2016-07-10 00:53:46.187851 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187850, event: psvc:dispatch, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  435. -150> 2016-07-10 00:53:46.187874 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187877 lease_expire=0.000000 has v0 lc 3914636
  436. -149> 2016-07-10 00:53:46.187900 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187900, event: logm:wait_for_readable, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  437. -148> 2016-07-10 00:53:46.187925 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187924, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  438. -147> 2016-07-10 00:53:46.187946 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187949 lease_expire=0.000000 has v0 lc 3914636
  439. -146> 2016-07-10 00:53:46.187971 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187970, event: paxos:wait_for_readable, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  440. -145> 2016-07-10 00:53:46.187996 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744896 ==== forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader v3 ==== 17414+0+0 (842647936 0 0) 0x5587f8210700 con 0x5587f8275780
  441. -144> 2016-07-10 00:53:46.188090 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188089, event: mon:_ms_dispatch, op: forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader
  442. -143> 2016-07-10 00:53:46.188119 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188119, event: mon:dispatch_op, op: forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader
  443. -142> 2016-07-10 00:53:46.188160 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188159, event: mon:_ms_dispatch, op: pg_stats(30 pgs tid 131362 v 0)
  444. -141> 2016-07-10 00:53:46.188184 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188183, event: mon:dispatch_op, op: pg_stats(30 pgs tid 131362 v 0)
  445. -140> 2016-07-10 00:53:46.188204 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188204, event: psvc:dispatch, op: pg_stats(30 pgs tid 131362 v 0)
  446. -139> 2016-07-10 00:53:46.188222 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188225 lease_expire=0.000000 has v0 lc 3914636
  447. -138> 2016-07-10 00:53:46.188248 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188247, event: pgmap:wait_for_readable, op: pg_stats(30 pgs tid 131362 v 0)
  448. -137> 2016-07-10 00:53:46.188268 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188268, event: pgmap:wait_for_readable/paxos, op: pg_stats(30 pgs tid 131362 v 0)
  449. -136> 2016-07-10 00:53:46.188286 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188289 lease_expire=0.000000 has v0 lc 3914636
  450. -135> 2016-07-10 00:53:46.188309 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188309, event: paxos:wait_for_readable, op: pg_stats(30 pgs tid 131362 v 0)
  451. -134> 2016-07-10 00:53:46.188351 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188331, event: done, op: forward(??? ) to leader
  452. -133> 2016-07-10 00:53:46.188391 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733962 ==== forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader v3 ==== 10056+0+0 (2547666369 0 0) 0x5587f8210100 con 0x5587f8275600
  453. -132> 2016-07-10 00:53:46.188435 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188434, event: mon:_ms_dispatch, op: forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader
  454. -131> 2016-07-10 00:53:46.188465 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188464, event: mon:dispatch_op, op: forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader
  455. -130> 2016-07-10 00:53:46.188502 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188501, event: mon:_ms_dispatch, op: pg_stats(17 pgs tid 131361 v 0)
  456. -129> 2016-07-10 00:53:46.188524 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188524, event: mon:dispatch_op, op: pg_stats(17 pgs tid 131361 v 0)
  457. -128> 2016-07-10 00:53:46.188547 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188547, event: psvc:dispatch, op: pg_stats(17 pgs tid 131361 v 0)
  458. -127> 2016-07-10 00:53:46.188566 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188569 lease_expire=0.000000 has v0 lc 3914636
  459. -126> 2016-07-10 00:53:46.188591 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188590, event: pgmap:wait_for_readable, op: pg_stats(17 pgs tid 131361 v 0)
  460. -125> 2016-07-10 00:53:46.188612 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188611, event: pgmap:wait_for_readable/paxos, op: pg_stats(17 pgs tid 131361 v 0)
  461. -124> 2016-07-10 00:53:46.188630 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188633 lease_expire=0.000000 has v0 lc 3914636
  462. -123> 2016-07-10 00:53:46.188655 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188654, event: paxos:wait_for_readable, op: pg_stats(17 pgs tid 131361 v 0)
  463. -122> 2016-07-10 00:53:46.188677 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188677, event: done, op: forward(??? ) to leader
  464. -121> 2016-07-10 00:53:46.188713 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744897 ==== forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader v3 ==== 11754+0+0 (493424996 0 0) 0x5587f8210a00 con 0x5587f8275780
  465. -120> 2016-07-10 00:53:46.188758 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188757, event: mon:_ms_dispatch, op: forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader
  466. -119> 2016-07-10 00:53:46.188787 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188786, event: mon:dispatch_op, op: forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader
  467. -118> 2016-07-10 00:53:46.188824 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188823, event: mon:_ms_dispatch, op: pg_stats(20 pgs tid 439 v 0)
  468. -117> 2016-07-10 00:53:46.188847 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188846, event: mon:dispatch_op, op: pg_stats(20 pgs tid 439 v 0)
  469. -116> 2016-07-10 00:53:46.188867 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188866, event: psvc:dispatch, op: pg_stats(20 pgs tid 439 v 0)
  470. -115> 2016-07-10 00:53:46.188885 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188888 lease_expire=0.000000 has v0 lc 3914636
  471. -114> 2016-07-10 00:53:46.188911 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188910, event: pgmap:wait_for_readable, op: pg_stats(20 pgs tid 439 v 0)
  472. -113> 2016-07-10 00:53:46.188931 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188930, event: pgmap:wait_for_readable/paxos, op: pg_stats(20 pgs tid 439 v 0)
  473. -112> 2016-07-10 00:53:46.188948 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188951 lease_expire=0.000000 has v0 lc 3914636
  474. -111> 2016-07-10 00:53:46.188973 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188971, event: paxos:wait_for_readable, op: pg_stats(20 pgs tid 439 v 0)
  475. -110> 2016-07-10 00:53:46.188995 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188994, event: done, op: forward(??? ) to leader
  476. -109> 2016-07-10 00:53:46.189041 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744898 ==== forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader v3 ==== 434+0+0 (4293513980 0 0) 0x5587f8210d00 con 0x5587f8275780
  477. -108> 2016-07-10 00:53:46.189086 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189085, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader
  478. -107> 2016-07-10 00:53:46.189115 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189115, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader
  479. -106> 2016-07-10 00:53:46.189148 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189147, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 416 v 0)
  480. -105> 2016-07-10 00:53:46.189172 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189171, event: mon:dispatch_op, op: pg_stats(0 pgs tid 416 v 0)
  481. -104> 2016-07-10 00:53:46.189194 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189194, event: psvc:dispatch, op: pg_stats(0 pgs tid 416 v 0)
  482. -103> 2016-07-10 00:53:46.189213 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.189216 lease_expire=0.000000 has v0 lc 3914636
  483. -102> 2016-07-10 00:53:46.189239 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189238, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 416 v 0)
  484. -101> 2016-07-10 00:53:46.189259 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189259, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 416 v 0)
  485. -100> 2016-07-10 00:53:46.189352 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.189355 lease_expire=0.000000 has v0 lc 3914636
  486. -99> 2016-07-10 00:53:46.189379 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189379, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 416 v 0)
  487. -98> 2016-07-10 00:53:46.189403 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189402, event: done, op: forward(??? ) to leader
  488. -97> 2016-07-10 00:53:46.189521 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733964 ==== paxos(last lc 3914636 fc 3914095 pn 4600 opn 0) v3 ==== 80+0+0 (1503394860 0 0) 0x5587f861ed00 con 0x5587f8275600
  489. -96> 2016-07-10 00:53:46.189569 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189568, event: mon:_ms_dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  490. -95> 2016-07-10 00:53:46.189595 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189595, event: mon:dispatch_op, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  491. -94> 2016-07-10 00:53:46.189622 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189622, event: paxos:dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  492. -93> 2016-07-10 00:53:46.189691 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189690, event: paxos:handle_last, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  493. -92> 2016-07-10 00:53:46.189722 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189722, event: done, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  494. -91> 2016-07-10 00:53:46.189829 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733965 ==== time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 ) v1 ==== 36+0+0 (640517914 0 0) 0x5587f829f200 con 0x5587f8275600
  495. -90> 2016-07-10 00:53:46.189882 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189881, event: mon:_ms_dispatch, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  496. -89> 2016-07-10 00:53:46.189911 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189910, event: mon:dispatch_op, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  497. -88> 2016-07-10 00:53:46.189952 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189951, event: done, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  498. -87> 2016-07-10 00:53:46.193677 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744899 ==== mon_metadata magic: 0 v1 ==== 394+0+0 (2086889972 0 0) 0x5587f82ea800 con 0x5587f8275780
  499. -86> 2016-07-10 00:53:46.193724 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.193723, event: mon:_ms_dispatch, op: mon_metadata magic: 0
  500. -85> 2016-07-10 00:53:46.193748 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.193748, event: mon:dispatch_op, op: mon_metadata magic: 0
  501. -84> 2016-07-10 00:53:46.194073 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.194073, event: done, op: mon_metadata magic: 0
  502. -83> 2016-07-10 00:53:46.197015 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744900 ==== paxos(last lc 3914636 fc 3914095 pn 4600 opn 0) v3 ==== 80+0+0 (1594699587 0 0) 0x5587f861ef80 con 0x5587f8275780
  503. -82> 2016-07-10 00:53:46.197084 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197083, event: mon:_ms_dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  504. -81> 2016-07-10 00:53:46.197106 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197104, event: mon:dispatch_op, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  505. -80> 2016-07-10 00:53:46.197133 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197133, event: paxos:dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  506. -79> 2016-07-10 00:53:46.197158 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197157, event: paxos:handle_last, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  507. -78> 2016-07-10 00:53:46.197200 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- paxos(lease lc 3914636 fc 3914095 pn 0 opn 0) v3 -- ?+0 0x5587f861ed00
  508. -77> 2016-07-10 00:53:46.197257 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- paxos(lease lc 3914636 fc 3914095 pn 0 opn 0) v3 -- ?+0 0x5587f861ea80
  509. -76> 2016-07-10 00:53:46.198494 7fdd8f5c3700 0 log_channel(cluster) log [INF] : monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  510. -75> 2016-07-10 00:53:46.198514 7fdd8f5c3700 10 log_client _send_to_monlog to self
  511. -74> 2016-07-10 00:53:46.198519 7fdd8f5c3700 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
  512. -73> 2016-07-10 00:53:46.198527 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198510 mon.0 10.0.0.11:6789/0 5 : cluster [INF] monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  513. -72> 2016-07-10 00:53:46.198553 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 5 at 2016-07-10 00:53:46.198510) v1 -- ?+0 0x5587f82a0640 con 0x5587f8274e80
  514. -71> 2016-07-10 00:53:46.198644 7fdd8f5c3700 0 log_channel(cluster) log [INF] : pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  515. -70> 2016-07-10 00:53:46.198659 7fdd8f5c3700 10 log_client _send_to_monlog to self
  516. -69> 2016-07-10 00:53:46.198665 7fdd8f5c3700 10 log_client log_queue is 6 last_log 6 sent 5 num 6 unsent 1 sending 1
  517. -68> 2016-07-10 00:53:46.198672 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198656 mon.0 10.0.0.11:6789/0 6 : cluster [INF] pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  518. -67> 2016-07-10 00:53:46.198719 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 6 at 2016-07-10 00:53:46.198656) v1 -- ?+0 0x5587f82a0880 con 0x5587f8274e80
  519. -66> 2016-07-10 00:53:46.198847 7fdd8f5c3700 0 log_channel(cluster) log [INF] : fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  520. -65> 2016-07-10 00:53:46.198863 7fdd8f5c3700 10 log_client _send_to_monlog to self
  521. -64> 2016-07-10 00:53:46.198868 7fdd8f5c3700 10 log_client log_queue is 7 last_log 7 sent 6 num 7 unsent 1 sending 1
  522. -63> 2016-07-10 00:53:46.198876 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198861 mon.0 10.0.0.11:6789/0 7 : cluster [INF] fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  523. -62> 2016-07-10 00:53:46.198898 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 7 at 2016-07-10 00:53:46.198861) v1 -- ?+0 0x5587f82a0ac0 con 0x5587f8274e80
  524. -61> 2016-07-10 00:53:46.199056 7fdd8f5c3700 0 log_channel(cluster) log [INF] : osdmap e474: 16 osds: 16 up, 16 in
  525. -60> 2016-07-10 00:53:46.199071 7fdd8f5c3700 10 log_client _send_to_monlog to self
  526. -59> 2016-07-10 00:53:46.199076 7fdd8f5c3700 10 log_client log_queue is 8 last_log 8 sent 7 num 8 unsent 1 sending 1
  527. -58> 2016-07-10 00:53:46.199082 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.199068 mon.0 10.0.0.11:6789/0 8 : cluster [INF] osdmap e474: 16 osds: 16 up, 16 in
RAW Paste Data