API tools faq
paste
Guest User

Untitled

a guest
Jul 9th, 2016
177
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 69.52 KB | None | 0 0
raw download clone embed print report
  1. -8> 2016-07-10 00:53:45.503560 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503560, event: callback finished, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  2. -7> 2016-07-10 00:53:45.503585 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503585, event: psvc:dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  3. -6> 2016-07-10 00:53:45.503607 7f0c31c6b700 5 mon.cephmon1@0(leader).paxos(paxos active c 3914095..3914636) is_readable = 1 - now=2016-07-10 00:53:45.503610 lease_expire=2016-07-10 00:53:50.499732 has v0 lc 3914636
  4. -5> 2016-07-10 00:53:45.503637 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503636, event: osdmap:preprocess_query, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  5. -4> 2016-07-10 00:53:45.503661 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503661, event: osdmap:preprocess_command, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  6. -3> 2016-07-10 00:53:45.503776 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503775, event: osdmap:prepare_update, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  7. -2> 2016-07-10 00:53:45.503816 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503815, event: osdmap:prepare_command, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  8. -1> 2016-07-10 00:53:45.503916 7f0c31c6b700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:45.503915, event: osdmap:prepare_command_impl, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  9. 0> 2016-07-10 00:53:45.511588 7f0c31c6b700 -1 *** Caught signal (Segmentation fault) **
  10. in thread 7f0c31c6b700 thread_name:ms_dispatch
  11.  
  12. ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
  13. 1: (()+0x5055ea) [0x564f430df5ea]
  14. 2: (()+0xf100) [0x7f0c391ed100]
  15. 3: (OSDMonitor::prepare_command_pool_set(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x122f) [0x564f42eeb97f]
  16. 4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&)+0xf02c) [0x564f42efb68c]
  17. 5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x64f) [0x564f42f0046f]
  18. 6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x307) [0x564f42f01fc7]
  19. 7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xe0b) [0x564f42eb060b]
  20. 8: (PaxosService::C_RetryMessage::_finish(int)+0x74) [0x564f42eb3db4]
  21. 9: (C_MonOp::finish(int)+0x79) [0x564f42e7e5b9]
  22. 10: (Context::complete(int)+0x9) [0x564f42e7d5f9]
  23. 11: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xac) [0x564f42e83e4c]
  24. 12: (Paxos::finish_round()+0x11e) [0x564f42ea802e]
  25. 13: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xef2) [0x564f42ea94d2]
  26. 14: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x304) [0x564f42ea9ec4]
  27. 15: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xb8a) [0x564f42e789ca]
  28. 16: (Monitor::_ms_dispatch(Message*)+0x6c9) [0x564f42e79519]
  29. 17: (Monitor::ms_dispatch(Message*)+0x23) [0x564f42e9a063]
  30. 18: (DispatchQueue::entry()+0x78a) [0x564f432f2d1a]
  31. 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x564f431e820d]
  32. 20: (()+0x7dc5) [0x7f0c391e5dc5]
  33. 21: (clone()+0x6d) [0x7f0c37aadced]
  34. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  35.  
  36. --- logging levels ---
  37. 0/ 5 none
  38. 0/ 1 lockdep
  39. 0/ 1 context
  40. 1/ 1 crush
  41. 1/ 5 mds
  42. 1/ 5 mds_balancer
  43. 1/ 5 mds_locker
  44. 1/ 5 mds_log
  45. 1/ 5 mds_log_expire
  46. 1/ 5 mds_migrator
  47. 0/ 1 buffer
  48. 0/ 1 timer
  49. 0/ 1 filer
  50. 0/ 1 striper
  51. 0/ 1 objecter
  52. 0/ 5 rados
  53. 0/ 5 rbd
  54. 0/ 5 rbd_mirror
  55. 0/ 5 rbd_replay
  56. 0/ 5 journaler
  57. 0/ 5 objectcacher
  58. 0/ 5 client
  59. 0/ 5 osd
  60. 0/ 5 optracker
  61. 0/ 5 objclass
  62. 1/ 3 filestore
  63. 1/ 3 journal
  64. 0/ 5 ms
  65. 1/ 5 mon
  66. 0/10 monc
  67. 1/ 5 paxos
  68. 0/ 5 tp
  69. 1/ 5 auth
  70. 1/ 5 crypto
  71. 1/ 1 finisher
  72. 1/ 5 heartbeatmap
  73. 1/ 5 perfcounter
  74. 1/ 5 rgw
  75. 1/10 civetweb
  76. 1/ 5 javaclient
  77. 1/ 5 asok
  78. 1/ 1 throttle
  79. 0/ 0 refs
  80. 1/ 5 xio
  81. 1/ 5 compressor
  82. 1/ 5 newstore
  83. 1/ 5 bluestore
  84. 1/ 5 bluefs
  85. 1/ 3 bdev
  86. 1/ 5 kstore
  87. 4/ 5 rocksdb
  88. 4/ 5 leveldb
  89. 1/ 5 kinetic
  90. 1/ 5 fuse
  91. -2/-2 (syslog threshold)
  92. -1/-1 (stderr threshold)
  93. max_recent 10000
  94. max_new 1000
  95. log_file /var/log/ceph/ceph-mon.cephmon1.log
  96. --- end dump of recent events ---
  97. 2016-07-10 00:53:45.876504 7fdd97da64c0 0 set uid:gid to 167:167 (ceph:ceph)
  98. 2016-07-10 00:53:45.876574 7fdd97da64c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mon, pid 17937
  99. 2016-07-10 00:53:45.876683 7fdd97da64c0 0 pidfile_write: ignore empty --pid-file
  100. 2016-07-10 00:53:46.021225 7fdd97da64c0 1 leveldb: Recovering log #525160
  101. 2016-07-10 00:53:46.021398 7fdd97da64c0 1 leveldb: Level-0 table #525162: started
  102. 2016-07-10 00:53:46.023255 7fdd97da64c0 1 leveldb: Level-0 table #525162: 192 bytes OK
  103. 2016-07-10 00:53:46.025897 7fdd97da64c0 1 leveldb: Delete type=0 #525160
  104.  
  105. 2016-07-10 00:53:46.026171 7fdd97da64c0 1 leveldb: Delete type=3 #525158
  106.  
  107. 2016-07-10 00:53:46.027545 7fdd97da64c0 0 starting mon.cephmon1 rank 0 at 10.0.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon1 fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  108. 2016-07-10 00:53:46.028204 7fdd97da64c0 1 mon.cephmon1@-1(probing) e1 preinit fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  109. 2016-07-10 00:53:46.029412 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(pgmap 1943096..1943707) refresh upgraded, format 0 -> 1
  110. 2016-07-10 00:53:46.029449 7fdd97da64c0 1 mon.cephmon1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
  111. 2016-07-10 00:53:46.085370 7fdd97da64c0 0 mon.cephmon1@-1(probing).mds e46 print_map
  112. e46
  113. enable_multiple, ever_enabled_multiple: 0,0
  114. compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  115.  
  116. Filesystem 'ceph-gen2' (2)
  117. fs_name ceph-gen2
  118. epoch 43
  119. flags 0
  120. created 2016-06-11 21:53:02.142649
  121. modified 2016-06-14 11:09:16.783356
  122. tableserver 0
  123. root 0
  124. session_timeout 60
  125. session_autoclose 300
  126. max_file_size 1099511627776
  127. last_failure 0
  128. last_failure_osd_epoch 216
  129. compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  130. max_mds 1
  131. in 0
  132. up {0=54100}
  133. failed
  134. damaged
  135. stopped
  136. data_pools 4
  137. metadata_pool 5
  138. inline_data disabled
  139. 54100: 10.0.0.12:6800/12411 'cephmon2' mds.0.39 up:active seq 7
  140.  
  141.  
  142. Standby daemons:
  143.  
  144. 44097: 10.0.0.11:6800/5378 'cephmon1' mds.-1.0 up:standby seq 539373
  145. 54103: 10.0.0.13:6800/12460 'cephmon3' mds.-1.0 up:standby seq 539371
  146.  
  147. 2016-07-10 00:53:46.086399 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  148. 2016-07-10 00:53:46.086423 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  149. 2016-07-10 00:53:46.086438 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  150. 2016-07-10 00:53:46.086448 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  151. 2016-07-10 00:53:46.087714 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(auth 751..888) refresh upgraded, format 0 -> 1
  152. 2016-07-10 00:53:46.092721 7fdd97da64c0 0 mon.cephmon1@-1(probing) e1 my rank is now 0 (was -1)
  153. 2016-07-10 00:53:46.095724 7fdd8d3bd700 0 -- 10.0.0.11:6789/0 >> 10.0.0.12:6789/0 pipe(0x5587f8676000 sd=11 :37630 s=2 pgs=2493 cs=1 l=0 c=0x5587f8275600).reader missed message? skipped from seq 0 to 699733957
  154. 2016-07-10 00:53:46.096136 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1 calling new monitor election
  155. 2016-07-10 00:53:46.096868 7fdd8f5c3700 1 mon.cephmon1@0(electing).elector(90) init, last seen epoch 90
  156. 2016-07-10 00:53:46.096850 7fdd8d2bc700 0 -- 10.0.0.11:6789/0 >> 10.0.0.13:6789/0 pipe(0x5587f8677400 sd=12 :43136 s=2 pgs=3635 cs=1 l=0 c=0x5587f8275780).reader missed message? skipped from seq 0 to 233744893
  157. 2016-07-10 00:53:46.107284 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1@0 won leader election with quorum 0,1,2
  158. 2016-07-10 00:53:46.109361 7fdd8f5c3700 0 log_channel(cluster) log [INF] : HEALTH_OK
  159. 2016-07-10 00:53:46.187279 7fdd8f5c3700 0 mon.cephmon1@0(leader) e1 handle_command mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1
  160. 2016-07-10 00:53:46.187410 7fdd8f5c3700 0 log_channel(audit) log [INF] : from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  161. 2016-07-10 00:53:46.198494 7fdd8f5c3700 0 log_channel(cluster) log [INF] : monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  162. 2016-07-10 00:53:46.198644 7fdd8f5c3700 0 log_channel(cluster) log [INF] : pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  163. 2016-07-10 00:53:46.198847 7fdd8f5c3700 0 log_channel(cluster) log [INF] : fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  164. 2016-07-10 00:53:46.199056 7fdd8f5c3700 0 log_channel(cluster) log [INF] : osdmap e474: 16 osds: 16 up, 16 in
  165. 2016-07-10 00:53:46.208221 7fdd8f5c3700 -1 *** Caught signal (Segmentation fault) **
  166. in thread 7fdd8f5c3700 thread_name:ms_dispatch
  167.  
  168. ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
  169. 1: (()+0x5055ea) [0x5587ec4995ea]
  170. 2: (()+0xf100) [0x7fdd96b45100]
  171. 3: (OSDMonitor::prepare_command_pool_set(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x122f) [0x5587ec2a597f]
  172. 4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&)+0xf02c) [0x5587ec2b568c]
  173. 5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x64f) [0x5587ec2ba46f]
  174. 6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x307) [0x5587ec2bbfc7]
  175. 7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xe0b) [0x5587ec26a60b]
  176. 8: (PaxosService::C_RetryMessage::_finish(int)+0x74) [0x5587ec26ddb4]
  177. 9: (C_MonOp::finish(int)+0x79) [0x5587ec2385b9]
  178. 10: (Context::complete(int)+0x9) [0x5587ec2375f9]
  179. 11: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xac) [0x5587ec23de4c]
  180. 12: (Paxos::finish_round()+0x11e) [0x5587ec26202e]
  181. 13: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xef2) [0x5587ec2634d2]
  182. 14: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x304) [0x5587ec263ec4]
  183. 15: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xb8a) [0x5587ec2329ca]
  184. 16: (Monitor::_ms_dispatch(Message*)+0x6c9) [0x5587ec233519]
  185. 17: (Monitor::ms_dispatch(Message*)+0x23) [0x5587ec254063]
  186. 18: (DispatchQueue::entry()+0x78a) [0x5587ec6acd1a]
  187. 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x5587ec5a220d]
  188. 20: (()+0x7dc5) [0x7fdd96b3ddc5]
  189. 21: (clone()+0x6d) [0x7fdd95405ced]
  190. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  191.  
  192. --- begin dump of recent events ---
  193. -355> 2016-07-10 00:53:45.851883 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perfcounters_dump hook 0x5587f81ca030
  194. -354> 2016-07-10 00:53:45.851939 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 1 hook 0x5587f81ca030
  195. -353> 2016-07-10 00:53:45.851960 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf dump hook 0x5587f81ca030
  196. -352> 2016-07-10 00:53:45.851979 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perfcounters_schema hook 0x5587f81ca030
  197. -351> 2016-07-10 00:53:45.851996 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 2 hook 0x5587f81ca030
  198. -350> 2016-07-10 00:53:45.852013 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf schema hook 0x5587f81ca030
  199. -349> 2016-07-10 00:53:45.852053 7fdd97da64c0 5 asok(0x5587f82a2000) register_command perf reset hook 0x5587f81ca030
  200. -348> 2016-07-10 00:53:45.852065 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config show hook 0x5587f81ca030
  201. -347> 2016-07-10 00:53:45.852077 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config set hook 0x5587f81ca030
  202. -346> 2016-07-10 00:53:45.852097 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config get hook 0x5587f81ca030
  203. -345> 2016-07-10 00:53:45.852117 7fdd97da64c0 5 asok(0x5587f82a2000) register_command config diff hook 0x5587f81ca030
  204. -344> 2016-07-10 00:53:45.852408 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log flush hook 0x5587f81ca030
  205. -343> 2016-07-10 00:53:45.852431 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log dump hook 0x5587f81ca030
  206. -342> 2016-07-10 00:53:45.852443 7fdd97da64c0 5 asok(0x5587f82a2000) register_command log reopen hook 0x5587f81ca030
  207. -341> 2016-07-10 00:53:45.876504 7fdd97da64c0 0 set uid:gid to 167:167 (ceph:ceph)
  208. -340> 2016-07-10 00:53:45.876574 7fdd97da64c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mon, pid 17937
  209. -339> 2016-07-10 00:53:45.876683 7fdd97da64c0 0 pidfile_write: ignore empty --pid-file
  210. -338> 2016-07-10 00:53:45.993655 7fdd97da64c0 5 asok(0x5587f82a2000) init /var/run/ceph/ceph-mon.cephmon1.asok
  211. -337> 2016-07-10 00:53:45.993689 7fdd97da64c0 5 asok(0x5587f82a2000) bind_and_listen /var/run/ceph/ceph-mon.cephmon1.asok
  212. -336> 2016-07-10 00:53:45.993991 7fdd97da64c0 5 asok(0x5587f82a2000) register_command 0 hook 0x5587f81c6090
  213. -335> 2016-07-10 00:53:45.994016 7fdd97da64c0 5 asok(0x5587f82a2000) register_command version hook 0x5587f81c6090
  214. -334> 2016-07-10 00:53:45.994058 7fdd97da64c0 5 asok(0x5587f82a2000) register_command git_version hook 0x5587f81c6090
  215. -333> 2016-07-10 00:53:45.994080 7fdd97da64c0 5 asok(0x5587f82a2000) register_command help hook 0x5587f81ca130
  216. -332> 2016-07-10 00:53:45.994093 7fdd97da64c0 5 asok(0x5587f82a2000) register_command get_command_descriptions hook 0x5587f81ca140
  217. -331> 2016-07-10 00:53:45.994330 7fdd92f27700 5 asok(0x5587f82a2000) entry start
  218. -330> 2016-07-10 00:53:46.021225 7fdd97da64c0 1 leveldb: Recovering log #525160
  219. -329> 2016-07-10 00:53:46.021398 7fdd97da64c0 1 leveldb: Level-0 table #525162: started
  220. -328> 2016-07-10 00:53:46.023255 7fdd97da64c0 1 leveldb: Level-0 table #525162: 192 bytes OK
  221. -327> 2016-07-10 00:53:46.025897 7fdd97da64c0 1 leveldb: Delete type=0 #525160
  222.  
  223. -326> 2016-07-10 00:53:46.026171 7fdd97da64c0 1 leveldb: Delete type=3 #525158
  224.  
  225. -325> 2016-07-10 00:53:46.027545 7fdd97da64c0 0 starting mon.cephmon1 rank 0 at 10.0.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon1 fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  226. -324> 2016-07-10 00:53:46.027648 7fdd97da64c0 1 -- 10.0.0.11:6789/0 learned my addr 10.0.0.11:6789/0
  227. -323> 2016-07-10 00:53:46.027668 7fdd97da64c0 1 accepter.accepter.bind my_inst.addr is 10.0.0.11:6789/0 need_addr=0
  228. -322> 2016-07-10 00:53:46.027805 7fdd97da64c0 5 adding auth protocol: cephx
  229. -321> 2016-07-10 00:53:46.027825 7fdd97da64c0 5 adding auth protocol: cephx
  230. -320> 2016-07-10 00:53:46.027886 7fdd97da64c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  231. -319> 2016-07-10 00:53:46.027914 7fdd97da64c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  232. -318> 2016-07-10 00:53:46.028204 7fdd97da64c0 1 mon.cephmon1@-1(probing) e1 preinit fsid a8171427-141c-4766-9e0f-533d86dd4ef8
  233. -317> 2016-07-10 00:53:46.029412 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(pgmap 1943096..1943707) refresh upgraded, format 0 -> 1
  234. -316> 2016-07-10 00:53:46.029449 7fdd97da64c0 1 mon.cephmon1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
  235. -315> 2016-07-10 00:53:46.085345 7fdd97da64c0 4 mon.cephmon1@-1(probing).mds e46 new map
  236. -314> 2016-07-10 00:53:46.085370 7fdd97da64c0 0 mon.cephmon1@-1(probing).mds e46 print_map
  237. e46
  238. enable_multiple, ever_enabled_multiple: 0,0
  239. compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  240.  
  241. Filesystem 'ceph-gen2' (2)
  242. fs_name ceph-gen2
  243. epoch 43
  244. flags 0
  245. created 2016-06-11 21:53:02.142649
  246. modified 2016-06-14 11:09:16.783356
  247. tableserver 0
  248. root 0
  249. session_timeout 60
  250. session_autoclose 300
  251. max_file_size 1099511627776
  252. last_failure 0
  253. last_failure_osd_epoch 216
  254. compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
  255. max_mds 1
  256. in 0
  257. up {0=54100}
  258. failed
  259. damaged
  260. stopped
  261. data_pools 4
  262. metadata_pool 5
  263. inline_data disabled
  264. 54100: 10.0.0.12:6800/12411 'cephmon2' mds.0.39 up:active seq 7
  265.  
  266.  
  267. Standby daemons:
  268.  
  269. 44097: 10.0.0.11:6800/5378 'cephmon1' mds.-1.0 up:standby seq 539373
  270. 54103: 10.0.0.13:6800/12460 'cephmon3' mds.-1.0 up:standby seq 539371
  271.  
  272. -313> 2016-07-10 00:53:46.086399 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  273. -312> 2016-07-10 00:53:46.086423 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  274. -311> 2016-07-10 00:53:46.086438 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  275. -310> 2016-07-10 00:53:46.086448 7fdd97da64c0 0 mon.cephmon1@-1(probing).osd e474 crush map has features 2234490552320, adjusting msgr requires
  276. -309> 2016-07-10 00:53:46.087714 7fdd97da64c0 1 mon.cephmon1@-1(probing).paxosservice(auth 751..888) refresh upgraded, format 0 -> 1
  277. -308> 2016-07-10 00:53:46.092072 7fdd97da64c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cephmon1/keyring
  278. -307> 2016-07-10 00:53:46.092096 7fdd97da64c0 5 asok(0x5587f82a2000) register_command mon_status hook 0x5587f81ca5f0
  279. -306> 2016-07-10 00:53:46.092112 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum_status hook 0x5587f81ca5f0
  280. -305> 2016-07-10 00:53:46.092124 7fdd97da64c0 5 asok(0x5587f82a2000) register_command sync_force hook 0x5587f81ca5f0
  281. -304> 2016-07-10 00:53:46.092143 7fdd97da64c0 5 asok(0x5587f82a2000) register_command add_bootstrap_peer_hint hook 0x5587f81ca5f0
  282. -303> 2016-07-10 00:53:46.092162 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum enter hook 0x5587f81ca5f0
  283. -302> 2016-07-10 00:53:46.092180 7fdd97da64c0 5 asok(0x5587f82a2000) register_command quorum exit hook 0x5587f81ca5f0
  284. -301> 2016-07-10 00:53:46.092199 7fdd97da64c0 5 asok(0x5587f82a2000) register_command ops hook 0x5587f81ca5f0
  285. -300> 2016-07-10 00:53:46.092270 7fdd97da64c0 1 -- 10.0.0.11:6789/0 messenger.start
  286. -299> 2016-07-10 00:53:46.092408 7fdd97da64c0 2 mon.cephmon1@-1(probing) e1 init
  287. -298> 2016-07-10 00:53:46.092643 7fdd97da64c0 1 accepter.accepter.start
  288. -297> 2016-07-10 00:53:46.092721 7fdd97da64c0 0 mon.cephmon1@-1(probing) e1 my rank is now 0 (was -1)
  289. -296> 2016-07-10 00:53:46.092739 7fdd97da64c0 1 -- 10.0.0.11:6789/0 mark_down_all
  290. -295> 2016-07-10 00:53:46.092771 7fdd97da64c0 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- mon_probe(probe a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon1) v6 -- ?+0 0x5587f861ed00
  291. -294> 2016-07-10 00:53:46.092941 7fdd97da64c0 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- mon_probe(probe a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon1) v6 -- ?+0 0x5587f861ef80
  292. -293> 2016-07-10 00:53:46.095724 7fdd8d3bd700 0 -- 10.0.0.11:6789/0 >> 10.0.0.12:6789/0 pipe(0x5587f8676000 sd=11 :37630 s=2 pgs=2493 cs=1 l=0 c=0x5587f8275600).reader missed message? skipped from seq 0 to 699733957
  293. -292> 2016-07-10 00:53:46.095876 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733957 ==== mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 )) v6 ==== 567+0+0 (3214913024 0 0) 0x5587f861ed00 con 0x5587f8275600
  294. -291> 2016-07-10 00:53:46.095951 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.095950, event: mon:_ms_dispatch, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  295. -290> 2016-07-10 00:53:46.095990 7fdd8f5c3700 5 mon.cephmon1@0(probing) e1 _ms_dispatch setting monitor caps on this connection
  296. -289> 2016-07-10 00:53:46.096016 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.096015, event: mon:dispatch_op, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  297. -288> 2016-07-10 00:53:46.096136 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1 calling new monitor election
  298. -287> 2016-07-10 00:53:46.096154 7fdd8f5c3700 10 log_client _send_to_monlog to self
  299. -286> 2016-07-10 00:53:46.096160 7fdd8f5c3700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
  300. -285> 2016-07-10 00:53:46.096167 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.096151 mon.0 10.0.0.11:6789/0 1 : cluster [INF] mon.cephmon1 calling new monitor election
  301. -284> 2016-07-10 00:53:46.096204 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 1 at 2016-07-10 00:53:46.096151) v1 -- ?+0 0x5587f829f680 con 0x5587f8274e80
  302. -283> 2016-07-10 00:53:46.096284 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(0) start -- can i be leader?
  303. -282> 2016-07-10 00:53:46.096868 7fdd8f5c3700 1 mon.cephmon1@0(electing).elector(90) init, last seen epoch 90
  304. -281> 2016-07-10 00:53:46.096850 7fdd8d2bc700 0 -- 10.0.0.11:6789/0 >> 10.0.0.13:6789/0 pipe(0x5587f8677400 sd=12 :43136 s=2 pgs=3635 cs=1 l=0 c=0x5587f8275780).reader missed message? skipped from seq 0 to 233744893
  305. -280> 2016-07-10 00:53:46.100941 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 propose 91) v5 -- ?+0 0x5587f8327700
  306. -279> 2016-07-10 00:53:46.101063 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 propose 91) v5 -- ?+0 0x5587f83279c0
  307. -278> 2016-07-10 00:53:46.101122 7fdd8f5c3700 5 -- op tracker -- seq: 1, time: 2016-07-10 00:53:46.101120, event: done, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon2 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  308. -277> 2016-07-10 00:53:46.101186 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 1 at 2016-07-10 00:53:46.096151) v1 ==== 0+0+0 (0 0 0) 0x5587f829f680 con 0x5587f8274e80
  309. -276> 2016-07-10 00:53:46.101238 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101238, event: mon:_ms_dispatch, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  310. -275> 2016-07-10 00:53:46.101261 7fdd8f5c3700 5 mon.cephmon1@0(electing) e1 _ms_dispatch setting monitor caps on this connection
  311. -274> 2016-07-10 00:53:46.101276 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101276, event: mon:dispatch_op, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  312. -273> 2016-07-10 00:53:46.101304 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101303, event: psvc:dispatch, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  313. -272> 2016-07-10 00:53:46.101323 7fdd8f5c3700 5 mon.cephmon1@0(electing).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.101326 lease_expire=0.000000 has v0 lc 3914636
  314. -271> 2016-07-10 00:53:46.101355 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101355, event: logm:wait_for_readable, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  315. -270> 2016-07-10 00:53:46.101382 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101382, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  316. -269> 2016-07-10 00:53:46.101407 7fdd8f5c3700 5 mon.cephmon1@0(electing).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.101410 lease_expire=0.000000 has v0 lc 3914636
  317. -268> 2016-07-10 00:53:46.101437 7fdd8f5c3700 5 -- op tracker -- seq: 2, time: 2016-07-10 00:53:46.101435, event: paxos:wait_for_readable, op: log(1 entries from seq 1 at 2016-07-10 00:53:46.096151)
  318. -267> 2016-07-10 00:53:46.101470 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744893 ==== mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 )) v6 ==== 567+0+0 (2951463228 0 0) 0x5587f861ef80 con 0x5587f8275780
  319. -266> 2016-07-10 00:53:46.101543 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101542, event: mon:_ms_dispatch, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  320. -265> 2016-07-10 00:53:46.101575 7fdd8f5c3700 5 mon.cephmon1@0(electing) e1 _ms_dispatch setting monitor caps on this connection
  321. -264> 2016-07-10 00:53:46.101596 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101596, event: mon:dispatch_op, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  322. -263> 2016-07-10 00:53:46.101642 7fdd8f5c3700 5 -- op tracker -- seq: 3, time: 2016-07-10 00:53:46.101641, event: done, op: mon_probe(reply a8171427-141c-4766-9e0f-533d86dd4ef8 name cephmon3 quorum 0,1,2 paxos( fc 3914095 lc 3914636 ))
  323. -262> 2016-07-10 00:53:46.106174 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744894 ==== election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91) v5 ==== 31040+0+0 (2406017146 0 0) 0x5587f83279c0 con 0x5587f8275780
  324. -261> 2016-07-10 00:53:46.106236 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106233, event: mon:_ms_dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  325. -260> 2016-07-10 00:53:46.106266 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106266, event: mon:dispatch_op, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  326. -259> 2016-07-10 00:53:46.106294 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106293, event: elector:dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  327. -258> 2016-07-10 00:53:46.106345 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106344, event: elector:handle_ack, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  328. -257> 2016-07-10 00:53:46.106368 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) handle_ack from mon.2
  329. -256> 2016-07-10 00:53:46.106381 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) so far i have {0=576460752032874495,2=576460752032874495}
  330. -255> 2016-07-10 00:53:46.106401 7fdd8f5c3700 5 -- op tracker -- seq: 4, time: 2016-07-10 00:53:46.106400, event: done, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  331. -254> 2016-07-10 00:53:46.106487 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733958 ==== election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91) v5 ==== 31040+0+0 (2406017146 0 0) 0x5587f8327700 con 0x5587f8275600
  332. -253> 2016-07-10 00:53:46.106532 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106531, event: mon:_ms_dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  333. -252> 2016-07-10 00:53:46.106557 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106557, event: mon:dispatch_op, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  334. -251> 2016-07-10 00:53:46.106582 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106582, event: elector:dispatch, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  335. -250> 2016-07-10 00:53:46.106623 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.106622, event: elector:handle_ack, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  336. -249> 2016-07-10 00:53:46.106644 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) handle_ack from mon.1
  337. -248> 2016-07-10 00:53:46.106654 7fdd8f5c3700 5 mon.cephmon1@0(electing).elector(91) so far i have {0=576460752032874495,1=576460752032874495,2=576460752032874495}
  338. -247> 2016-07-10 00:53:46.107157 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 victory 92) v5 -- ?+0 0x5587f8327c80
  339. -246> 2016-07-10 00:53:46.107219 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- election(a8171427-141c-4766-9e0f-533d86dd4ef8 victory 92) v5 -- ?+0 0x5587f8327180
  340. -245> 2016-07-10 00:53:46.107284 7fdd8f5c3700 0 log_channel(cluster) log [INF] : mon.cephmon1@0 won leader election with quorum 0,1,2
  341. -244> 2016-07-10 00:53:46.107306 7fdd8f5c3700 10 log_client _send_to_monlog to self
  342. -243> 2016-07-10 00:53:46.107311 7fdd8f5c3700 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
  343. -242> 2016-07-10 00:53:46.107319 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.107300 mon.0 10.0.0.11:6789/0 2 : cluster [INF] mon.cephmon1@0 won leader election with quorum 0,1,2
  344. -241> 2016-07-10 00:53:46.107346 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 2 at 2016-07-10 00:53:46.107300) v1 -- ?+0 0x5587f829fb00 con 0x5587f8274e80
  345. -240> 2016-07-10 00:53:46.107901 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- paxos(collect lc 3914636 fc 3914095 pn 4600 opn 0) v3 -- ?+0 0x5587f861ed00
  346. -239> 2016-07-10 00:53:46.107952 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- paxos(collect lc 3914636 fc 3914095 pn 4600 opn 0) v3 -- ?+0 0x5587f861ef80
  347. -238> 2016-07-10 00:53:46.108083 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- time_check( ping e 92 r 1 ) v1 -- ?+0 0x5587f829f440
  348. -237> 2016-07-10 00:53:46.108134 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- time_check( ping e 92 r 1 ) v1 -- ?+0 0x5587f829fd40
  349. -236> 2016-07-10 00:53:46.109361 7fdd8f5c3700 0 log_channel(cluster) log [INF] : HEALTH_OK
  350. -235> 2016-07-10 00:53:46.109383 7fdd8f5c3700 10 log_client _send_to_monlog to self
  351. -234> 2016-07-10 00:53:46.109388 7fdd8f5c3700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
  352. -233> 2016-07-10 00:53:46.109396 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.109379 mon.0 10.0.0.11:6789/0 3 : cluster [INF] HEALTH_OK
  353. -232> 2016-07-10 00:53:46.109442 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 3 at 2016-07-10 00:53:46.109379) v1 -- ?+0 0x5587f82a0400 con 0x5587f8274e80
  354. -231> 2016-07-10 00:53:46.184528 7fdd8f5c3700 5 -- op tracker -- seq: 5, time: 2016-07-10 00:53:46.184526, event: done, op: election(a8171427-141c-4766-9e0f-533d86dd4ef8 ack 91)
  355. -230> 2016-07-10 00:53:46.184622 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 2 at 2016-07-10 00:53:46.107300) v1 ==== 0+0+0 (0 0 0) 0x5587f829fb00 con 0x5587f8274e80
  356. -229> 2016-07-10 00:53:46.184692 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184691, event: mon:_ms_dispatch, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  357. -228> 2016-07-10 00:53:46.184715 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184713, event: mon:dispatch_op, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  358. -227> 2016-07-10 00:53:46.184740 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184740, event: psvc:dispatch, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  359. -226> 2016-07-10 00:53:46.184765 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.184768 lease_expire=0.000000 has v0 lc 3914636
  360. -225> 2016-07-10 00:53:46.184797 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184797, event: logm:wait_for_readable, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  361. -224> 2016-07-10 00:53:46.184823 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184822, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  362. -223> 2016-07-10 00:53:46.184846 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.184849 lease_expire=0.000000 has v0 lc 3914636
  363. -222> 2016-07-10 00:53:46.184865 7fdd8f5c3700 5 -- op tracker -- seq: 6, time: 2016-07-10 00:53:46.184864, event: paxos:wait_for_readable, op: log(1 entries from seq 2 at 2016-07-10 00:53:46.107300)
  364. -221> 2016-07-10 00:53:46.184892 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733961 ==== forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader v3 ==== 638+0+0 (2780614426 0 0) 0x5587f820fe00 con 0x5587f8275600
  365. -220> 2016-07-10 00:53:46.184942 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.184942, event: mon:_ms_dispatch, op: forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader
  366. -219> 2016-07-10 00:53:46.184973 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.184972, event: mon:dispatch_op, op: forward(log(1 entries from seq 39 at 2016-07-10 00:53:44.550142) v1 caps allow * tid 391 con_features 0) to leader
  367. -218> 2016-07-10 00:53:46.185090 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185090, event: mon:_ms_dispatch, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  368. -217> 2016-07-10 00:53:46.185121 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185121, event: mon:dispatch_op, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  369. -216> 2016-07-10 00:53:46.185148 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185146, event: psvc:dispatch, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  370. -215> 2016-07-10 00:53:46.185170 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185177 lease_expire=0.000000 has v0 lc 3914636
  371. -214> 2016-07-10 00:53:46.185203 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185201, event: logm:wait_for_readable, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  372. -213> 2016-07-10 00:53:46.185227 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185227, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  373. -212> 2016-07-10 00:53:46.185251 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185254 lease_expire=0.000000 has v0 lc 3914636
  374. -211> 2016-07-10 00:53:46.185276 7fdd8f5c3700 5 -- op tracker -- seq: 8, time: 2016-07-10 00:53:46.185275, event: paxos:wait_for_readable, op: log(1 entries from seq 39 at 2016-07-10 00:53:44.550142)
  375. -210> 2016-07-10 00:53:46.185301 7fdd8f5c3700 5 -- op tracker -- seq: 7, time: 2016-07-10 00:53:46.185301, event: done, op: forward(??? ) to leader
  376. -209> 2016-07-10 00:53:46.185342 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 3 at 2016-07-10 00:53:46.109379) v1 ==== 0+0+0 (0 0 0) 0x5587f82a0400 con 0x5587f8274e80
  377. -208> 2016-07-10 00:53:46.185387 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185387, event: mon:_ms_dispatch, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  378. -207> 2016-07-10 00:53:46.185413 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185413, event: mon:dispatch_op, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  379. -206> 2016-07-10 00:53:46.185437 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185437, event: psvc:dispatch, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  380. -205> 2016-07-10 00:53:46.185460 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185463 lease_expire=0.000000 has v0 lc 3914636
  381. -204> 2016-07-10 00:53:46.185486 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185485, event: logm:wait_for_readable, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  382. -203> 2016-07-10 00:53:46.185511 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185509, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  383. -202> 2016-07-10 00:53:46.185533 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185536 lease_expire=0.000000 has v0 lc 3914636
  384. -201> 2016-07-10 00:53:46.185558 7fdd8f5c3700 5 -- op tracker -- seq: 9, time: 2016-07-10 00:53:46.185557, event: paxos:wait_for_readable, op: log(1 entries from seq 3 at 2016-07-10 00:53:46.109379)
  385. -200> 2016-07-10 00:53:46.185584 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733959 ==== forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader v3 ==== 12320+0+0 (2403987416 0 0) 0x5587f820f500 con 0x5587f8275600
  386. -199> 2016-07-10 00:53:46.185628 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185627, event: mon:_ms_dispatch, op: forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader
  387. -198> 2016-07-10 00:53:46.185684 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185656, event: mon:dispatch_op, op: forward(pg_stats(21 pgs tid 131361 v 0) v1 caps allow profile osd tid 388 con_features 576460752032874495) to leader
  388. -197> 2016-07-10 00:53:46.185733 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185732, event: mon:_ms_dispatch, op: pg_stats(21 pgs tid 131361 v 0)
  389. -196> 2016-07-10 00:53:46.185757 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185756, event: mon:dispatch_op, op: pg_stats(21 pgs tid 131361 v 0)
  390. -195> 2016-07-10 00:53:46.185777 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185777, event: psvc:dispatch, op: pg_stats(21 pgs tid 131361 v 0)
  391. -194> 2016-07-10 00:53:46.185795 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185798 lease_expire=0.000000 has v0 lc 3914636
  392. -193> 2016-07-10 00:53:46.185820 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185820, event: pgmap:wait_for_readable, op: pg_stats(21 pgs tid 131361 v 0)
  393. -192> 2016-07-10 00:53:46.185841 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185840, event: pgmap:wait_for_readable/paxos, op: pg_stats(21 pgs tid 131361 v 0)
  394. -191> 2016-07-10 00:53:46.185858 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.185861 lease_expire=0.000000 has v0 lc 3914636
  395. -190> 2016-07-10 00:53:46.185882 7fdd8f5c3700 5 -- op tracker -- seq: 11, time: 2016-07-10 00:53:46.185881, event: paxos:wait_for_readable, op: pg_stats(21 pgs tid 131361 v 0)
  396. -189> 2016-07-10 00:53:46.185903 7fdd8f5c3700 5 -- op tracker -- seq: 10, time: 2016-07-10 00:53:46.185903, event: done, op: forward(??? ) to leader
  397. -188> 2016-07-10 00:53:46.185935 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744895 ==== forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader v3 ==== 2132+0+0 (4169066638 0 0) 0x5587f820f200 con 0x5587f8275780
  398. -187> 2016-07-10 00:53:46.185978 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.185977, event: mon:_ms_dispatch, op: forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader
  399. -186> 2016-07-10 00:53:46.186007 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.186006, event: mon:dispatch_op, op: forward(pg_stats(3 pgs tid 131363 v 0) v1 caps allow profile osd tid 988091 con_features 576460752032874495) to leader
  400. -185> 2016-07-10 00:53:46.186063 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186063, event: mon:_ms_dispatch, op: pg_stats(3 pgs tid 131363 v 0)
  401. -184> 2016-07-10 00:53:46.186087 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186086, event: mon:dispatch_op, op: pg_stats(3 pgs tid 131363 v 0)
  402. -183> 2016-07-10 00:53:46.186108 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186107, event: psvc:dispatch, op: pg_stats(3 pgs tid 131363 v 0)
  403. -182> 2016-07-10 00:53:46.186200 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.186204 lease_expire=0.000000 has v0 lc 3914636
  404. -181> 2016-07-10 00:53:46.186234 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186234, event: pgmap:wait_for_readable, op: pg_stats(3 pgs tid 131363 v 0)
  405. -180> 2016-07-10 00:53:46.186256 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186255, event: pgmap:wait_for_readable/paxos, op: pg_stats(3 pgs tid 131363 v 0)
  406. -179> 2016-07-10 00:53:46.186279 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.186282 lease_expire=0.000000 has v0 lc 3914636
  407. -178> 2016-07-10 00:53:46.186306 7fdd8f5c3700 5 -- op tracker -- seq: 13, time: 2016-07-10 00:53:46.186305, event: paxos:wait_for_readable, op: pg_stats(3 pgs tid 131363 v 0)
  408. -177> 2016-07-10 00:53:46.186328 7fdd8f5c3700 5 -- op tracker -- seq: 12, time: 2016-07-10 00:53:46.186327, event: done, op: forward(??? ) to leader
  409. -176> 2016-07-10 00:53:46.186366 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733963 ==== mon_metadata magic: 0 v1 ==== 414+0+0 (2638027890 0 0) 0x5587f82ea800 con 0x5587f8275600
  410. -175> 2016-07-10 00:53:46.186408 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186407, event: mon:_ms_dispatch, op: mon_metadata magic: 0
  411. -174> 2016-07-10 00:53:46.186431 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186430, event: mon:dispatch_op, op: mon_metadata magic: 0
  412. -173> 2016-07-10 00:53:46.186861 7fdd8f5c3700 5 -- op tracker -- seq: 14, time: 2016-07-10 00:53:46.186860, event: done, op: mon_metadata magic: 0
  413. -172> 2016-07-10 00:53:46.186909 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733960 ==== forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader v3 ==== 383+0+0 (3863550525 0 0) 0x5587f820f800 con 0x5587f8275600
  414. -171> 2016-07-10 00:53:46.186955 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.186954, event: mon:_ms_dispatch, op: forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader
  415. -170> 2016-07-10 00:53:46.186985 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.186983, event: mon:dispatch_op, op: forward(mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1 caps allow * tid 390 con_features 576460752032874495) to leader
  416. -169> 2016-07-10 00:53:46.187048 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187048, event: mon:_ms_dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  417. -168> 2016-07-10 00:53:46.187073 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187073, event: mon:dispatch_op, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  418. -167> 2016-07-10 00:53:46.187279 7fdd8f5c3700 0 mon.cephmon1@0(leader) e1 handle_command mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0) v1
  419. -166> 2016-07-10 00:53:46.187410 7fdd8f5c3700 0 log_channel(audit) log [INF] : from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  420. -165> 2016-07-10 00:53:46.187442 7fdd8f5c3700 10 log_client _send_to_monlog to self
  421. -164> 2016-07-10 00:53:46.187449 7fdd8f5c3700 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
  422. -163> 2016-07-10 00:53:46.187477 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.187434 mon.0 10.0.0.11:6789/0 4 : audit [INF] from='client.144115 :/0' entity='unknown.' cmd=[{"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"}]: dispatch
  423. -162> 2016-07-10 00:53:46.187516 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 4 at 2016-07-10 00:53:46.187434) v1 -- ?+0 0x5587f82a01c0 con 0x5587f8274e80
  424. -161> 2016-07-10 00:53:46.187566 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187565, event: psvc:dispatch, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  425. -160> 2016-07-10 00:53:46.187589 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187592 lease_expire=0.000000 has v0 lc 3914636
  426. -159> 2016-07-10 00:53:46.187618 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187616, event: osdmap:wait_for_readable, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  427. -158> 2016-07-10 00:53:46.187641 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187640, event: osdmap:wait_for_readable/paxos, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  428. -157> 2016-07-10 00:53:46.187660 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187663 lease_expire=0.000000 has v0 lc 3914636
  429. -156> 2016-07-10 00:53:46.187684 7fdd8f5c3700 5 -- op tracker -- seq: 16, time: 2016-07-10 00:53:46.187684, event: paxos:wait_for_readable, op: mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "lxc", "val": "2"} v 0)
  430. -155> 2016-07-10 00:53:46.187716 7fdd8f5c3700 5 -- op tracker -- seq: 15, time: 2016-07-10 00:53:46.187715, event: done, op: forward(??? ) to leader
  431. -154> 2016-07-10 00:53:46.187755 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.0 10.0.0.11:6789/0 0 ==== log(1 entries from seq 4 at 2016-07-10 00:53:46.187434) v1 ==== 0+0+0 (0 0 0) 0x5587f82a01c0 con 0x5587f8274e80
  432. -153> 2016-07-10 00:53:46.187800 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187799, event: mon:_ms_dispatch, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  433. -152> 2016-07-10 00:53:46.187826 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187826, event: mon:dispatch_op, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  434. -151> 2016-07-10 00:53:46.187851 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187850, event: psvc:dispatch, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  435. -150> 2016-07-10 00:53:46.187874 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187877 lease_expire=0.000000 has v0 lc 3914636
  436. -149> 2016-07-10 00:53:46.187900 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187900, event: logm:wait_for_readable, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  437. -148> 2016-07-10 00:53:46.187925 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187924, event: logm:wait_for_readable/paxos, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  438. -147> 2016-07-10 00:53:46.187946 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.187949 lease_expire=0.000000 has v0 lc 3914636
  439. -146> 2016-07-10 00:53:46.187971 7fdd8f5c3700 5 -- op tracker -- seq: 17, time: 2016-07-10 00:53:46.187970, event: paxos:wait_for_readable, op: log(1 entries from seq 4 at 2016-07-10 00:53:46.187434)
  440. -145> 2016-07-10 00:53:46.187996 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744896 ==== forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader v3 ==== 17414+0+0 (842647936 0 0) 0x5587f8210700 con 0x5587f8275780
  441. -144> 2016-07-10 00:53:46.188090 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188089, event: mon:_ms_dispatch, op: forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader
  442. -143> 2016-07-10 00:53:46.188119 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188119, event: mon:dispatch_op, op: forward(pg_stats(30 pgs tid 131362 v 0) v1 caps allow profile osd tid 988092 con_features 576460752032874495) to leader
  443. -142> 2016-07-10 00:53:46.188160 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188159, event: mon:_ms_dispatch, op: pg_stats(30 pgs tid 131362 v 0)
  444. -141> 2016-07-10 00:53:46.188184 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188183, event: mon:dispatch_op, op: pg_stats(30 pgs tid 131362 v 0)
  445. -140> 2016-07-10 00:53:46.188204 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188204, event: psvc:dispatch, op: pg_stats(30 pgs tid 131362 v 0)
  446. -139> 2016-07-10 00:53:46.188222 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188225 lease_expire=0.000000 has v0 lc 3914636
  447. -138> 2016-07-10 00:53:46.188248 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188247, event: pgmap:wait_for_readable, op: pg_stats(30 pgs tid 131362 v 0)
  448. -137> 2016-07-10 00:53:46.188268 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188268, event: pgmap:wait_for_readable/paxos, op: pg_stats(30 pgs tid 131362 v 0)
  449. -136> 2016-07-10 00:53:46.188286 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188289 lease_expire=0.000000 has v0 lc 3914636
  450. -135> 2016-07-10 00:53:46.188309 7fdd8f5c3700 5 -- op tracker -- seq: 19, time: 2016-07-10 00:53:46.188309, event: paxos:wait_for_readable, op: pg_stats(30 pgs tid 131362 v 0)
  451. -134> 2016-07-10 00:53:46.188351 7fdd8f5c3700 5 -- op tracker -- seq: 18, time: 2016-07-10 00:53:46.188331, event: done, op: forward(??? ) to leader
  452. -133> 2016-07-10 00:53:46.188391 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733962 ==== forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader v3 ==== 10056+0+0 (2547666369 0 0) 0x5587f8210100 con 0x5587f8275600
  453. -132> 2016-07-10 00:53:46.188435 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188434, event: mon:_ms_dispatch, op: forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader
  454. -131> 2016-07-10 00:53:46.188465 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188464, event: mon:dispatch_op, op: forward(pg_stats(17 pgs tid 131361 v 0) v1 caps allow profile osd tid 392 con_features 576460752032874495) to leader
  455. -130> 2016-07-10 00:53:46.188502 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188501, event: mon:_ms_dispatch, op: pg_stats(17 pgs tid 131361 v 0)
  456. -129> 2016-07-10 00:53:46.188524 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188524, event: mon:dispatch_op, op: pg_stats(17 pgs tid 131361 v 0)
  457. -128> 2016-07-10 00:53:46.188547 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188547, event: psvc:dispatch, op: pg_stats(17 pgs tid 131361 v 0)
  458. -127> 2016-07-10 00:53:46.188566 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188569 lease_expire=0.000000 has v0 lc 3914636
  459. -126> 2016-07-10 00:53:46.188591 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188590, event: pgmap:wait_for_readable, op: pg_stats(17 pgs tid 131361 v 0)
  460. -125> 2016-07-10 00:53:46.188612 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188611, event: pgmap:wait_for_readable/paxos, op: pg_stats(17 pgs tid 131361 v 0)
  461. -124> 2016-07-10 00:53:46.188630 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188633 lease_expire=0.000000 has v0 lc 3914636
  462. -123> 2016-07-10 00:53:46.188655 7fdd8f5c3700 5 -- op tracker -- seq: 21, time: 2016-07-10 00:53:46.188654, event: paxos:wait_for_readable, op: pg_stats(17 pgs tid 131361 v 0)
  463. -122> 2016-07-10 00:53:46.188677 7fdd8f5c3700 5 -- op tracker -- seq: 20, time: 2016-07-10 00:53:46.188677, event: done, op: forward(??? ) to leader
  464. -121> 2016-07-10 00:53:46.188713 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744897 ==== forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader v3 ==== 11754+0+0 (493424996 0 0) 0x5587f8210a00 con 0x5587f8275780
  465. -120> 2016-07-10 00:53:46.188758 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188757, event: mon:_ms_dispatch, op: forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader
  466. -119> 2016-07-10 00:53:46.188787 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188786, event: mon:dispatch_op, op: forward(pg_stats(20 pgs tid 439 v 0) v1 caps allow profile osd tid 988093 con_features 576460752032874495) to leader
  467. -118> 2016-07-10 00:53:46.188824 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188823, event: mon:_ms_dispatch, op: pg_stats(20 pgs tid 439 v 0)
  468. -117> 2016-07-10 00:53:46.188847 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188846, event: mon:dispatch_op, op: pg_stats(20 pgs tid 439 v 0)
  469. -116> 2016-07-10 00:53:46.188867 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188866, event: psvc:dispatch, op: pg_stats(20 pgs tid 439 v 0)
  470. -115> 2016-07-10 00:53:46.188885 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188888 lease_expire=0.000000 has v0 lc 3914636
  471. -114> 2016-07-10 00:53:46.188911 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188910, event: pgmap:wait_for_readable, op: pg_stats(20 pgs tid 439 v 0)
  472. -113> 2016-07-10 00:53:46.188931 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188930, event: pgmap:wait_for_readable/paxos, op: pg_stats(20 pgs tid 439 v 0)
  473. -112> 2016-07-10 00:53:46.188948 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.188951 lease_expire=0.000000 has v0 lc 3914636
  474. -111> 2016-07-10 00:53:46.188973 7fdd8f5c3700 5 -- op tracker -- seq: 23, time: 2016-07-10 00:53:46.188971, event: paxos:wait_for_readable, op: pg_stats(20 pgs tid 439 v 0)
  475. -110> 2016-07-10 00:53:46.188995 7fdd8f5c3700 5 -- op tracker -- seq: 22, time: 2016-07-10 00:53:46.188994, event: done, op: forward(??? ) to leader
  476. -109> 2016-07-10 00:53:46.189041 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744898 ==== forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader v3 ==== 434+0+0 (4293513980 0 0) 0x5587f8210d00 con 0x5587f8275780
  477. -108> 2016-07-10 00:53:46.189086 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189085, event: mon:_ms_dispatch, op: forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader
  478. -107> 2016-07-10 00:53:46.189115 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189115, event: mon:dispatch_op, op: forward(pg_stats(0 pgs tid 416 v 0) v1 caps allow profile osd tid 988094 con_features 576460752032874495) to leader
  479. -106> 2016-07-10 00:53:46.189148 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189147, event: mon:_ms_dispatch, op: pg_stats(0 pgs tid 416 v 0)
  480. -105> 2016-07-10 00:53:46.189172 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189171, event: mon:dispatch_op, op: pg_stats(0 pgs tid 416 v 0)
  481. -104> 2016-07-10 00:53:46.189194 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189194, event: psvc:dispatch, op: pg_stats(0 pgs tid 416 v 0)
  482. -103> 2016-07-10 00:53:46.189213 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.189216 lease_expire=0.000000 has v0 lc 3914636
  483. -102> 2016-07-10 00:53:46.189239 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189238, event: pgmap:wait_for_readable, op: pg_stats(0 pgs tid 416 v 0)
  484. -101> 2016-07-10 00:53:46.189259 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189259, event: pgmap:wait_for_readable/paxos, op: pg_stats(0 pgs tid 416 v 0)
  485. -100> 2016-07-10 00:53:46.189352 7fdd8f5c3700 5 mon.cephmon1@0(leader).paxos(paxos recovering c 3914095..3914636) is_readable = 0 - now=2016-07-10 00:53:46.189355 lease_expire=0.000000 has v0 lc 3914636
  486. -99> 2016-07-10 00:53:46.189379 7fdd8f5c3700 5 -- op tracker -- seq: 25, time: 2016-07-10 00:53:46.189379, event: paxos:wait_for_readable, op: pg_stats(0 pgs tid 416 v 0)
  487. -98> 2016-07-10 00:53:46.189403 7fdd8f5c3700 5 -- op tracker -- seq: 24, time: 2016-07-10 00:53:46.189402, event: done, op: forward(??? ) to leader
  488. -97> 2016-07-10 00:53:46.189521 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733964 ==== paxos(last lc 3914636 fc 3914095 pn 4600 opn 0) v3 ==== 80+0+0 (1503394860 0 0) 0x5587f861ed00 con 0x5587f8275600
  489. -96> 2016-07-10 00:53:46.189569 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189568, event: mon:_ms_dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  490. -95> 2016-07-10 00:53:46.189595 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189595, event: mon:dispatch_op, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  491. -94> 2016-07-10 00:53:46.189622 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189622, event: paxos:dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  492. -93> 2016-07-10 00:53:46.189691 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189690, event: paxos:handle_last, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  493. -92> 2016-07-10 00:53:46.189722 7fdd8f5c3700 5 -- op tracker -- seq: 26, time: 2016-07-10 00:53:46.189722, event: done, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  494. -91> 2016-07-10 00:53:46.189829 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.1 10.0.0.12:6789/0 699733965 ==== time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 ) v1 ==== 36+0+0 (640517914 0 0) 0x5587f829f200 con 0x5587f8275600
  495. -90> 2016-07-10 00:53:46.189882 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189881, event: mon:_ms_dispatch, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  496. -89> 2016-07-10 00:53:46.189911 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189910, event: mon:dispatch_op, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  497. -88> 2016-07-10 00:53:46.189952 7fdd8f5c3700 5 -- op tracker -- seq: 27, time: 2016-07-10 00:53:46.189951, event: done, op: time_check( pong e 92 r 1 ts 2016-07-10 00:53:46.189703 )
  498. -87> 2016-07-10 00:53:46.193677 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744899 ==== mon_metadata magic: 0 v1 ==== 394+0+0 (2086889972 0 0) 0x5587f82ea800 con 0x5587f8275780
  499. -86> 2016-07-10 00:53:46.193724 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.193723, event: mon:_ms_dispatch, op: mon_metadata magic: 0
  500. -85> 2016-07-10 00:53:46.193748 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.193748, event: mon:dispatch_op, op: mon_metadata magic: 0
  501. -84> 2016-07-10 00:53:46.194073 7fdd8f5c3700 5 -- op tracker -- seq: 28, time: 2016-07-10 00:53:46.194073, event: done, op: mon_metadata magic: 0
  502. -83> 2016-07-10 00:53:46.197015 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 <== mon.2 10.0.0.13:6789/0 233744900 ==== paxos(last lc 3914636 fc 3914095 pn 4600 opn 0) v3 ==== 80+0+0 (1594699587 0 0) 0x5587f861ef80 con 0x5587f8275780
  503. -82> 2016-07-10 00:53:46.197084 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197083, event: mon:_ms_dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  504. -81> 2016-07-10 00:53:46.197106 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197104, event: mon:dispatch_op, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  505. -80> 2016-07-10 00:53:46.197133 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197133, event: paxos:dispatch, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  506. -79> 2016-07-10 00:53:46.197158 7fdd8f5c3700 5 -- op tracker -- seq: 29, time: 2016-07-10 00:53:46.197157, event: paxos:handle_last, op: paxos(last lc 3914636 fc 3914095 pn 4600 opn 0)
  507. -78> 2016-07-10 00:53:46.197200 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.1 10.0.0.12:6789/0 -- paxos(lease lc 3914636 fc 3914095 pn 0 opn 0) v3 -- ?+0 0x5587f861ed00
  508. -77> 2016-07-10 00:53:46.197257 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> mon.2 10.0.0.13:6789/0 -- paxos(lease lc 3914636 fc 3914095 pn 0 opn 0) v3 -- ?+0 0x5587f861ea80
  509. -76> 2016-07-10 00:53:46.198494 7fdd8f5c3700 0 log_channel(cluster) log [INF] : monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  510. -75> 2016-07-10 00:53:46.198514 7fdd8f5c3700 10 log_client _send_to_monlog to self
  511. -74> 2016-07-10 00:53:46.198519 7fdd8f5c3700 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
  512. -73> 2016-07-10 00:53:46.198527 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198510 mon.0 10.0.0.11:6789/0 5 : cluster [INF] monmap e1: 3 mons at {cephmon1=10.0.0.11:6789/0,cephmon2=10.0.0.12:6789/0,cephmon3=10.0.0.13:6789/0}
  513. -72> 2016-07-10 00:53:46.198553 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 5 at 2016-07-10 00:53:46.198510) v1 -- ?+0 0x5587f82a0640 con 0x5587f8274e80
  514. -71> 2016-07-10 00:53:46.198644 7fdd8f5c3700 0 log_channel(cluster) log [INF] : pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  515. -70> 2016-07-10 00:53:46.198659 7fdd8f5c3700 10 log_client _send_to_monlog to self
  516. -69> 2016-07-10 00:53:46.198665 7fdd8f5c3700 10 log_client log_queue is 6 last_log 6 sent 5 num 6 unsent 1 sending 1
  517. -68> 2016-07-10 00:53:46.198672 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198656 mon.0 10.0.0.11:6789/0 6 : cluster [INF] pgmap v1943707: 2304 pgs: 2304 active+clean; 12478 GB data, 24976 GB used, 6605 GB / 31581 GB avail
  518. -67> 2016-07-10 00:53:46.198719 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 6 at 2016-07-10 00:53:46.198656) v1 -- ?+0 0x5587f82a0880 con 0x5587f8274e80
  519. -66> 2016-07-10 00:53:46.198847 7fdd8f5c3700 0 log_channel(cluster) log [INF] : fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  520. -65> 2016-07-10 00:53:46.198863 7fdd8f5c3700 10 log_client _send_to_monlog to self
  521. -64> 2016-07-10 00:53:46.198868 7fdd8f5c3700 10 log_client log_queue is 7 last_log 7 sent 6 num 7 unsent 1 sending 1
  522. -63> 2016-07-10 00:53:46.198876 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.198861 mon.0 10.0.0.11:6789/0 7 : cluster [INF] fsmap e46: 1/1/1 up {0=cephmon2=up:active}, 2 up:standby
  523. -62> 2016-07-10 00:53:46.198898 7fdd8f5c3700 1 -- 10.0.0.11:6789/0 --> 10.0.0.11:6789/0 -- log(1 entries from seq 7 at 2016-07-10 00:53:46.198861) v1 -- ?+0 0x5587f82a0ac0 con 0x5587f8274e80
  524. -61> 2016-07-10 00:53:46.199056 7fdd8f5c3700 0 log_channel(cluster) log [INF] : osdmap e474: 16 osds: 16 up, 16 in
  525. -60> 2016-07-10 00:53:46.199071 7fdd8f5c3700 10 log_client _send_to_monlog to self
  526. -59> 2016-07-10 00:53:46.199076 7fdd8f5c3700 10 log_client log_queue is 8 last_log 8 sent 7 num 8 unsent 1 sending 1
  527. -58> 2016-07-10 00:53:46.199082 7fdd8f5c3700 10 log_client will send 2016-07-10 00:53:46.199068 mon.0 10.0.0.11:6789/0 8 : cluster [INF] osdmap e474: 16 osds: 16 up, 16 in
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!