API tools faq
paste
Advertisement
rzerres

ceph-mon-error-v.058-to-v0.59

rzerres
Mar 22nd, 2013
22
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 25.26 KB | None | 0 0
raw download clone embed print report
  1. -12> 2013-03-21 22:39:39.394142 7f354f167700 1 mon.1@2(peon).paxos(logm active c 2238948..2239448) is_readable now=2013-03-21 22:39:39.394144 lease_expire=2013-03-21 22:39:44.392271 has v0 lc 2239448
  2. -11> 2013-03-21 22:39:39.906733 7f354f167700 1 -- 172.16.0.4:6789/0 <== osd.0 172.16.0.3:6805/6356 340 ==== log(40 entries) v1 ==== 13278+0+0 (460620400 0 0) 0x2315600 con 0x24419a0
  3. -10> 2013-03-21 22:39:39.906795 7f354f167700 1 mon.1@2(peon).paxos(logm active c 2238948..2239448) is_readable now=2013-03-21 22:39:39.906797 lease_expire=2013-03-21 22:39:44.392271 has v0 lc 2239448
  4. -9> 2013-03-21 22:39:39.906947 7f354f167700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- forward(log(40 entries) v1) to leader v1 -- ?+0 0x2fda840
  5. -8> 2013-03-21 22:39:40.054247 7f354f167700 1 -- 172.16.0.4:6789/0 <== mon.0 172.16.0.2:6789/0 1400520731 ==== paxos(logm begin lc 2239448 fc 0 pn 19100 opn 0 gv {2239449=4325938}) v2 ==== 13357+0+0 (2839381263 0 0) 0x2fda840 con 0x22709a0
  6. -7> 2013-03-21 22:39:40.835084 7f354d559700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x3012780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 172.16.0.3:46986/0
  7. -6> 2013-03-21 22:39:40.835997 7f354d559700 2 -- 172.16.0.4:6789/0 >> 172.16.0.3:6789/0 pipe(0x3012780 sd=25 :6789 s=4 pgs=0 cs=0 l=0).fault 0: Success
  8. -5> 2013-03-21 22:39:41.771409 7f354f167700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- paxos(logm accept lc 2239448 fc 0 pn 19100 opn 0 gv {}) v2 -- ?+0 0x2db62c0
  9. -4> 2013-03-21 22:39:41.771554 7f354f167700 1 -- 172.16.0.4:6789/0 <== mds.0 172.16.0.3:6800/2067 267 ==== mdsbeacon(68226/0 up:replay seq 62877 v980) v2 ==== 279+0+0 (1052806109 0 0) 0x2de42c0 con 0x24416e0
  10. -3> 2013-03-21 22:39:41.771593 7f354f167700 1 mon.1@2(peon).paxos(mdsmap active c 1..980) is_readable now=2013-03-21 22:39:41.771595 lease_expire=2013-03-21 22:39:44.162517 has v980 lc 980
  11. -2> 2013-03-21 22:39:41.771673 7f354f167700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- forward(mdsbeacon(68226/0 up:replay seq 62877 v980) v2) to leader v1 -- ?+0 0x2fda840
  12. -1> 2013-03-21 22:39:41.814361 7f354f167700 1 -- 172.16.0.4:6789/0 <== mon.0 172.16.0.2:6789/0 1400520732 ==== paxos(logm commit lc 2239449 fc 0 pn 19100 opn 0 gv {2239449=4325938}) v2 ==== 13357+0+0 (2945232926 0 0) 0x2fda840 con 0x22709a0
  13. 0> 2013-03-21 22:39:41.817102 7f354f167700 -1 mon/MonitorStore.cc: In function 'void MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, const char*, bool)' thread 7f354f167700 time 2013-03-21 22:39:41.814607
  14. mon/MonitorStore.cc: 382: FAILED assert(!err)
  15.  
  16. ceph version 0.58 (ba3f91e7504867a52a83399d60917e3414e8c3e2)
  17. 1: (MonitorStore::write_bl_ss(ceph::buffer::list&, char const*, char const*, bool)+0xe18) [0x533d58]
  18. 2: (MonitorStore::put_bl_sn_map(char const*, std::_Rb_tree_iterator<std::pair<unsigned long const, ceph::buffer::list> >, std::_Rb_tree_iterator<std::pair<unsigned long const, ceph::buffer::list> >, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)+0x102) [0x534382]
  19. 3: (Paxos::store_state(MMonPaxos*)+0x2b5) [0x4a26d5]
  20. 4: (Paxos::handle_commit(MMonPaxos*)+0x2e5) [0x4a2f95]
  21. 5: (Paxos::dispatch(PaxosServiceMessage*)+0x2ab) [0x4a9ceb]
  22. 6: (Monitor::_ms_dispatch(Message*)+0x10e7) [0x48e057]
  23. 7: (Monitor::ms_dispatch(Message*)+0x32) [0x49eec2]
  24. 8: (DispatchQueue::entry()+0x341) [0x655d01]
  25. 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x5e3b4d]
  26. 10: (()+0x7e9a) [0x7f3554556e9a]
  27. 11: (clone()+0x6d) [0x7f35533f7cbd]
  28. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  29.  
  30. --- logging levels ---
  31. 0/ 5 none
  32. 0/ 1 lockdep
  33. 0/ 1 context
  34. 1/ 1 crush
  35. 1/ 5 mds
  36. 1/ 5 mds_balancer
  37. 1/ 5 mds_locker
  38. 1/ 5 mds_log
  39. 1/ 5 mds_log_expire
  40. 1/ 5 mds_migrator
  41. 0/ 1 buffer
  42. 0/ 1 timer
  43. 0/ 1 filer
  44. 0/ 1 striper
  45. 0/ 1 objecter
  46. 0/ 5 rados
  47. 0/ 5 rbd
  48. 0/ 5 journaler
  49. 0/ 5 objectcacher
  50. 0/ 5 client
  51. 0/ 5 osd
  52. 0/ 5 optracker
  53. 0/ 5 objclass
  54. 1/ 3 filestore
  55. 1/ 3 journal
  56. 0/ 5 ms
  57. 1/ 5 mon
  58. 0/10 monc
  59. 0/ 5 paxos
  60. 0/ 5 tp
  61. 1/ 5 auth
  62. 1/ 5 crypto
  63. 1/ 1 finisher
  64. 1/ 5 heartbeatmap
  65. 1/ 5 perfcounter
  66. 1/ 5 rgw
  67. 1/ 5 hadoop
  68. 1/ 5 javaclient
  69. 1/ 5 asok
  70. 1/ 1 throttle
  71. -2/-2 (syslog threshold)
  72. -1/-1 (stderr threshold)
  73. max_recent 100000
  74. max_new 1000
  75. log_file /var/log/ceph/mon.1.log
  76. --- end dump of recent events ---
  77. 2013-03-21 22:39:42.324268 7f354f167700 -1 *** Caught signal (Aborted) **
  78. in thread 7f354f167700
  79.  
  80.  
  81. 2013-03-21 22:56:23.908351 7f1e3103a780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 6524
  82. 2013-03-21 22:56:23.908834 7f1e3103a780 1 unable to open monitor store at /srv/ceph/mon.1
  83. 2013-03-21 22:56:23.908847 7f1e3103a780 1 check for old monitor store format
  84. 2013-03-21 22:56:23.908852 7f1e3103a780 1 store(/srv/ceph/mon.1) mount
  85. 2013-03-21 22:56:23.908877 7f1e3103a780 1 store(/srv/ceph/mon.1) mount
  86. 2013-03-21 22:56:23.908918 7f1e3103a780 1 found old GV monitor store format -- should convert!
  87. 2013-03-21 22:56:23.914932 7f1e3103a780 1 store(/srv/ceph/mon.1) mount
  88. 2013-03-21 22:57:36.779685 7f0eda64b780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 6678
  89.  
  90. root@dwssrv2:/var/log/ceph# ls -l /srv/ceph/mon.1
  91. insgesamt 24
  92. drwxr-xr-x 1 root root 276 Mär 21 22:31 auth
  93. drwxr-xr-x 1 root root 168 Mär 21 22:31 auth_gv
  94. -rw------- 1 root root 37 Dez 20 00:56 cluster_uuid
  95. -rw------- 1 root root 4 Mär 21 22:22 election_epoch
  96. -rw------- 1 root root 120 Dez 20 00:56 feature_set
  97. -rw------- 1 root root 2 Dez 20 00:59 joined
  98. -rw------- 1 root root 55 Dez 20 00:59 keyring
  99. -rw------- 1 root root 0 Dez 20 00:34 lock
  100. drwxr-xr-x 1 root root 7200 Mär 21 22:39 logm
  101. drwxr-xr-x 1 root root 7070 Mär 21 22:39 logm_gv
  102. -rw------- 1 root root 21 Dez 20 00:56 magic
  103. drwxr-xr-x 1 root root 5772 Mär 21 22:22 mdsmap
  104. drwxr-xr-x 1 root root 5070 Mär 21 22:22 mdsmap_gv
  105. drwxr-xr-x 1 root root 14 Jan 7 19:39 mkfs
  106. drwxr-xr-x 1 root root 116 Mär 21 22:22 monmap
  107. drwxr-xr-x 1 root root 6 Mär 13 11:15 monmap_gv
  108. drwxr-xr-x 1 root root 6828 Mär 21 22:22 osdmap
  109. drwxr-xr-x 1 root root 6720 Mär 19 01:25 osdmap_full
  110. drwxr-xr-x 1 root root 6720 Mär 19 01:25 osdmap_gv
  111. drwxr-xr-x 1 root root 7150 Mär 21 22:22 pgmap
  112. drwxr-xr-x 1 root root 7042 Mär 19 01:28 pgmap_gv
  113. drwxr-xr-x 1 root root 332 Mär 22 15:45 store.db
  114.  
  115. 2013-03-22 15:45:37.166785 7f41a22d5780 10 needs_conversion
  116. 2013-03-22 15:45:37.227624 7f41a22d5780 -1 ERROR: on disk data includes unsupported features: compat={},rocompat={},incompat={4=}
  117. 2013-03-22 16:16:20.800933 7f6948ca3780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 1221
  118. 6
  119. 2013-03-22 16:16:20.800947 7f6948ca3780 10 needs_conversion
  120. 2013-03-22 16:16:20.801386 7f6948ca3780 1 unable to open monitor store at /srv/ceph/mon.1
  121. 2013-03-22 16:16:20.801397 7f6948ca3780 1 check for old monitor store format
  122. 2013-03-22 16:16:20.801403 7f6948ca3780 1 store(/srv/ceph/mon.1) mount
  123. 2013-03-22 16:16:20.801428 7f6948ca3780 1 store(/srv/ceph/mon.1) mount
  124. 2013-03-22 16:16:20.801442 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl magic
  125. 2013-03-22 16:16:20.801449 7f6948ca3780 20 _check_gv_store
  126. 2013-03-22 16:16:20.801451 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl feature_set
  127. 2013-03-22 16:16:20.801460 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 120
  128. 2013-03-22 16:16:20.801473 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl feature_set = 120 bytes
  129. 2013-03-22 16:16:20.801487 7f6948ca3780 1 found old GV monitor store format -- should convert!
  130. 2013-03-22 16:16:20.807093 7f6948ca3780 1 store(/srv/ceph/mon.1) mount
  131. 2013-03-22 16:16:20.808295 7f6948ca3780 10 _convert_monitor
  132. 2013-03-22 16:16:20.808312 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl magic
  133. 2013-03-22 16:16:20.808332 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl keyring
  134. 2013-03-22 16:16:20.808340 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl feature_set
  135. 2013-03-22 16:16:20.808354 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl election_epoch
  136. 2013-03-22 16:16:20.808362 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl joined
  137. 2013-03-22 16:16:20.808402 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int joined = 1
  138. 2013-03-22 16:16:20.808422 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl magic
  139. 2013-03-22 16:16:20.808439 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 21
  140. 2013-03-22 16:16:20.808450 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl magic = 21 bytes
  141. 2013-03-22 16:16:20.808457 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl feature_set
  142. 2013-03-22 16:16:20.808472 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 120
  143. 2013-03-22 16:16:20.808480 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl feature_set = 120 bytes
  144. 2013-03-22 16:16:20.808486 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl cluster_uuid
  145. 2013-03-22 16:16:20.808515 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 37
  146. 2013-03-22 16:16:20.808524 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl cluster_uuid = 37 bytes
  147. 2013-03-22 16:16:20.808540 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int election_epoch = 268
  148. 2013-03-22 16:16:20.809517 7f6948ca3780 10 _convert_monitor finished
  149. 2013-03-22 16:16:20.809546 7f6948ca3780 10 _convert_machines
  150. 2013-03-22 16:16:20.809563 7f6948ca3780 10 _convert_machines auth
  151. 2013-03-22 16:16:20.809595 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth/first_committed = 5238
  152. 2013-03-22 16:16:20.809613 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth/last_committed = 5258
  153. 2013-03-22 16:16:20.809634 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth/accepted_pn = 19100
  154. 2013-03-22 16:16:20.809647 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth_gv
  155. 2013-03-22 16:16:20.809661 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth/5238
  156. 2013-03-22 16:16:20.809697 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 649
  157. 2013-03-22 16:16:20.809709 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl auth/5238 = 649 bytes
  158. 2013-03-22 16:16:20.809714 7f6948ca3780 20 _convert_machines auth ver 5238 bl 649
  159. 2013-03-22 16:16:20.809747 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth_gv/5238
  160. 2013-03-22 16:16:20.809776 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth_gv/5238 = 4313440
  161. 2013-03-22 16:16:20.809784 7f6948ca3780 20 _convert_machines auth ver 5238 -> 4313440
  162. 2013-03-22 16:16:20.810772 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth/5239
  163. 2013-03-22 16:16:20.810810 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 649
  164. 2013-03-22 16:16:20.810825 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl auth/5239 = 649 bytes
  165. 2013-03-22 16:16:20.810830 7f6948ca3780 20 _convert_machines auth ver 5239 bl 649
  166. 2013-03-22 16:16:20.810847 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth_gv/5239
  167. 2013-03-22 16:16:20.810873 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth_gv/5239 = 4314161
  168. 2013-03-22 16:16:20.810878 7f6948ca3780 20 _convert_machines auth ver 5239 -> 4314161
  169. 2013-03-22 16:16:20.811879 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth/5240
  170. 2013-03-22 16:16:20.811916 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 649
  171. 2013-03-22 16:16:20.811931 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl auth/5240 = 649 bytes
  172. 2013-03-22 16:16:20.811936 7f6948ca3780 20 _convert_machines auth ver 5240 bl 649
  173. 2013-03-22 16:16:20.811953 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth_gv/5240
  174. 2013-03-22 16:16:20.811979 7f6948ca3780 15 store(/srv/ceph/mon.1) get_int auth_gv/5240 = 4314883
  175. 2013-03-22 16:16:20.811985 7f6948ca3780 20 _convert_machines auth ver 5240 -> 4314883
  176. 2013-03-22 16:16:20.812941 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth/5241
  177. 2013-03-22 16:16:20.812974 7f6948ca3780 20 store(/srv/ceph/mon.1) reading at off 0 of 649
  178. 2013-03-22 16:16:20.812987 7f6948ca3780 15 store(/srv/ceph/mon.1) get_bl auth/5241 = 649 bytes
  179. 2013-03-22 16:16:20.812992 7f6948ca3780 20 _convert_machines auth ver 5241 bl 649
  180. 2013-03-22 16:16:20.813008 7f6948ca3780 15 store(/srv/ceph/mon.1) exists_bl auth_gv/5241
  181.  
  182. 2013-03-22 17:03:33.422422 7f9402fb2780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 13368
  183. 2013-03-22 17:03:33.422440 7f9402fb2780 10 needs_conversion
  184. 2013-03-22 17:03:33.478707 7f9402fb2780 -1 ERROR: on disk data includes unsupported features: compat={},rocompat={},incompat={4=}
  185. 2013-03-22 17:04:04.780241 7f1a1af81780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 13475
  186. 2013-03-22 17:04:04.780258 7f1a1af81780 10 needs_conversion
  187. 2013-03-22 17:04:04.798231 7f1a1af81780 -1 ERROR: on disk data includes unsupported features: compat={},rocompat={},incompat={4=}
  188.  
  189. 2013-03-22 17:10:37.311745 7ffa367c5780 10 mon.1@-1(probing).mds e980 update_logger
  190. 2013-03-22 17:10:37.311792 7ffa367c5780 15 mon.1@-1(probing).osd e0 update_from_paxos paxos e 4304, my e 0
  191. 2013-03-22 17:10:37.311853 7ffa367c5780 7 mon.1@-1(probing).osd e0 update_from_paxos loading latest full map e4304
  192. 2013-03-22 17:10:37.311957 7ffa367c5780 10 mon.1@-1(probing).osd e4304 adding osd.3 to down_pending_out map
  193. 2013-03-22 17:10:37.311968 7ffa367c5780 10 mon.1@-1(probing).osd e4304 adding osd.4 to down_pending_out map
  194. 2013-03-22 17:10:37.311971 7ffa367c5780 10 mon.1@-1(probing).osd e4304 adding osd.5 to down_pending_out map
  195. 2013-03-22 17:10:37.311973 7ffa367c5780 10 mon.1@-1(probing).osd e4304 adding osd.6 to down_pending_out map
  196. 2013-03-22 17:10:37.311977 7ffa367c5780 10 mon.1@-1(probing).osd e4304 send_to_waiting 4304
  197. 2013-03-22 17:10:37.311984 7ffa367c5780 10 mon.1@-1(probing).osd e4304 share_map_with_random_osd no up osd on our session map
  198. 2013-03-22 17:10:37.311986 7ffa367c5780 10 mon.1@-1(probing).osd e4304 update_logger
  199. 2013-03-22 17:10:37.311993 7ffa367c5780 0 mon.1@-1(probing).osd e4304 crush map has features 33816576, adjusting msgr requires
  200. 2013-03-22 17:10:37.311997 7ffa367c5780 0 mon.1@-1(probing).osd e4304 crush map has features 33816576, adjusting msgr requires
  201. 2013-03-22 17:10:37.311999 7ffa367c5780 0 mon.1@-1(probing).osd e4304 crush map has features 33816576, adjusting msgr requires
  202. 2013-03-22 17:10:37.312002 7ffa367c5780 0 mon.1@-1(probing).osd e4304 crush map has features 33816576, adjusting msgr requires
  203. 2013-03-22 17:10:37.312006 7ffa367c5780 10 mon.1@-1(probing).log v2239448 update_from_paxos
  204. 2013-03-22 17:10:37.312068 7ffa367c5780 10 mon.1@-1(probing).log v2239448 update_from_paxos version 2239448 summary v 0
  205. 2013-03-22 17:10:37.312097 7ffa367c5780 10 mon.1@-1(probing).log v2239448 update_from_paxos latest full 2239448
  206. 2013-03-22 17:10:37.312148 7ffa367c5780 7 mon.1@-1(probing).log v2239448 update_from_paxos loading summary e2239448
  207. 2013-03-22 17:10:37.312209 7ffa367c5780 7 mon.1@-1(probing).log v2239448 update_from_paxos loaded summary e2239448
  208. 2013-03-22 17:10:37.312228 7ffa367c5780 10 mon.1@-1(probing).log v2239448 check_subs
  209. 2013-03-22 17:10:37.312262 7ffa367c5780 10 mon.1@-1(probing).auth v5258 update_from_paxos
  210. 2013-03-22 17:10:37.315985 7ffa367c5780 10 mon.1@-1(probing).auth v5258 update_from_paxos version 5258 keys ver 0 latest 5258
  211. 2013-03-22 17:10:37.318398 7ffa367c5780 7 mon.1@-1(probing).auth v5258 update_from_paxos loading summary e 5258
  212. 2013-03-22 17:10:37.319617 7ffa367c5780 7 mon.1@-1(probing).auth v5258 update_from_paxos latest length 2002
  213. 2013-03-22 17:10:37.320889 7ffa367c5780 10 mon.1@-1(probing).auth v5258 update_from_paxos key server version 5258
  214. 2013-03-22 17:10:37.322089 7ffa367c5780 10 mon.1@-1(probing).auth v5258 update_from_paxos() last_allocated_id=68596 max_global_id=68596
  215. 2013-03-22 17:10:37.335037 7ffa367c5780 1 -- 172.16.0.4:6789/0 messenger.start
  216. 2013-03-22 17:10:37.335252 7ffa367c5780 2 mon.1@-1(probing) e4 init
  217. 2013-03-22 17:10:37.335503 7ffa367c5780 1 accepter.accepter.start
  218. 2013-03-22 17:10:37.335603 7ffa367c5780 10 mon.1@-1(probing) e4 bootstrap
  219. 2013-03-22 17:10:37.335625 7ffa367c5780 10 mon.1@-1(probing) e4 unregister_cluster_logger - not registered
  220. 2013-03-22 17:10:37.335634 7ffa367c5780 10 mon.1@-1(probing) e4 cancel_probe_timeout (none scheduled)
  221. 2013-03-22 17:10:37.335644 7ffa367c5780 0 mon.1@-1(probing) e4 my rank is now 2 (was -1)
  222. 2013-03-22 17:10:37.335654 7ffa367c5780 1 -- 172.16.0.4:6789/0 mark_down_all
  223. 2013-03-22 17:10:37.335679 7ffa367c5780 10 mon.1@2(probing) e4 reset_sync
  224. 2013-03-22 17:10:37.335687 7ffa367c5780 10 mon.1@2(probing) e4 reset
  225. 2013-03-22 17:10:37.335698 7ffa367c5780 10 mon.1@2(probing) e4 timecheck_finish
  226. 2013-03-22 17:10:37.335715 7ffa367c5780 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- canceling timeouts
  227. 2013-03-22 17:10:37.335762 7ffa367c5780 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- clearing queued proposals
  228. 2013-03-22 17:10:37.335780 7ffa367c5780 10 mon.1@2(probing).paxosservice(pgmap) restart
  229. 2013-03-22 17:10:37.335801 7ffa367c5780 10 mon.1@2(probing).paxosservice(mdsmap) restart
  230. 2013-03-22 17:10:37.335811 7ffa367c5780 10 mon.1@2(probing).paxosservice(osdmap) restart
  231. 2013-03-22 17:10:37.335819 7ffa367c5780 10 mon.1@2(probing).paxosservice(logm) restart
  232. 2013-03-22 17:10:37.335824 7ffa367c5780 10 mon.1@2(probing).paxosservice(monmap) restart
  233. 2013-03-22 17:10:37.335828 7ffa367c5780 10 mon.1@2(probing).paxosservice(auth) restart
  234. 2013-03-22 17:10:37.335836 7ffa367c5780 10 mon.1@2(probing) e4 cancel_probe_timeout (none scheduled)
  235. 2013-03-22 17:10:37.335843 7ffa367c5780 10 mon.1@2(probing) e4 reset_probe_timeout 0x18ae6b0 after 2 seconds
  236. 2013-03-22 17:10:37.335947 7ffa367c5780 10 mon.1@2(probing) e4 probing other monitors
  237. 2013-03-22 17:10:37.335971 7ffa367c5780 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- mon_probe(probe 3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b name 1) v4 -- ?+0 0x1c29900
  238. 2013-03-22 17:10:37.336127 7ffa367c5780 1 -- 172.16.0.4:6789/0 --> mon.1 172.16.0.3:6789/0 -- mon_probe(probe 3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b name 1) v4 -- ?+0 0x1c29000
  239. 2013-03-22 17:10:37.336722 7ffa2f760700 0 -- 172.16.0.4:6789/0 >> 172.16.0.3:6789/0 pipe(0x18cb280 sd=28 :0 s=1 pgs=0 cs=0 l=0).fault
  240. 2013-03-22 17:10:37.336723 7ffa2f861700 0 -- 172.16.0.4:6789/0 >> 172.16.0.2:6789/0 pipe(0x18cba00 sd=27 :0 s=1 pgs=0 cs=0 l=0).fault
  241. 2013-03-22 17:10:39.335973 7ffa31064700 4 mon.1@2(probing) e4 probe_timeout 0x18ae6b0
  242. 2013-03-22 17:10:39.336026 7ffa31064700 10 mon.1@2(probing) e4 bootstrap
  243. 2013-03-22 17:10:39.336032 7ffa31064700 10 mon.1@2(probing) e4 unregister_cluster_logger - not registered
  244. 2013-03-22 17:10:39.336037 7ffa31064700 10 mon.1@2(probing) e4 cancel_probe_timeout (none scheduled)
  245. 2013-03-22 17:10:39.336045 7ffa31064700 10 mon.1@2(probing) e4 reset_sync
  246. 2013-03-22 17:10:39.336056 7ffa31064700 10 mon.1@2(probing) e4 reset
  247. 2013-03-22 17:10:39.336061 7ffa31064700 10 mon.1@2(probing) e4 timecheck_finish
  248. 2013-03-22 17:10:39.336070 7ffa31064700 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- canceling timeouts
  249. 2013-03-22 17:10:39.336092 7ffa31064700 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- clearing queued proposals
  250. 2013-03-22 17:10:39.336105 7ffa31064700 10 mon.1@2(probing).paxosservice(pgmap) restart
  251. 2013-03-22 17:10:39.336112 7ffa31064700 10 mon.1@2(probing).paxosservice(mdsmap) restart
  252. 2013-03-22 17:10:39.336118 7ffa31064700 10 mon.1@2(probing).paxosservice(osdmap) restart
  253. 2013-03-22 17:10:39.336123 7ffa31064700 10 mon.1@2(probing).paxosservice(logm) restart
  254. 2013-03-22 17:10:39.336127 7ffa31064700 10 mon.1@2(probing).paxosservice(monmap) restart
  255. 2013-03-22 17:10:39.336130 7ffa31064700 10 mon.1@2(probing).paxosservice(auth) restart
  256. 2013-03-22 17:10:39.336135 7ffa31064700 10 mon.1@2(probing) e4 cancel_probe_timeout (none scheduled)
  257. 2013-03-22 17:10:39.336144 7ffa31064700 10 mon.1@2(probing) e4 reset_probe_timeout 0x18ae6d0 after 2 seconds
  258. 2013-03-22 17:10:39.336192 7ffa31064700 10 mon.1@2(probing) e4 probing other monitors
  259. 2013-03-22 17:10:39.336217 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- mon_probe(probe 3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b name 1) v4 -- ?+0 0x4341000
  260. 2013-03-22 17:10:39.336267 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.1 172.16.0.3:6789/0 -- mon_probe(probe 3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b name 1) v4 -- ?+0 0x4341600
  261. 2013-03-22 17:10:41.336228 7ffa31064700 4 mon.1@2(probing) e4 probe_timeout 0x18ae6d0
  262. 2013-03-22 17:10:41.336257 7ffa31064700 10 mon.1@2(probing) e4 bootstrap
  263. 2013-03-22 17:10:41.336265 7ffa31064700 10 mon.1@2(probing) e4 unregister_cluster_logger - not registered
  264. 2013-03-22 17:10:41.336270 7ffa31064700 10 mon.1@2(probing) e4 cancel_probe_timeout (none scheduled)
  265. 2013-03-22 17:10:41.336276 7ffa31064700 10 mon.1@2(probing) e4 reset_sync
  266. 2013-03-22 17:10:41.336281 7ffa31064700 10 mon.1@2(probing) e4 reset
  267. 2013-03-22 17:10:41.336286 7ffa31064700 10 mon.1@2(probing) e4 timecheck_finish
  268. 2013-03-22 17:10:41.336293 7ffa31064700 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- canceling timeouts
  269. 2013-03-22 17:10:41.336311 7ffa31064700 10 mon.1@2(probing).paxos(paxos recovering c 4325918..4325937) restart -- clearing queued proposals
  270. 2013-03-22 17:10:41.336323 7ffa31064700 10 mon.1@2(probing).paxosservice(pgmap) restart
  271. 2013-03-22 17:10:41.336328 7ffa31064700 10 mon.1@2(probing).paxosservice(mdsmap) restart
  272. 2013-03-22 17:10:41.336332 7ffa31064700 10 mon.1@2(probing).paxosservice(osdmap) restart
  273. 2013-03-22 17:10:41.336336 7ffa31064700 10 mon.1@2(probing).paxosservice(logm) restart
  274. 2013-03-22 17:10:41.336340 7ffa31064700 10 mon.1@2(probing).paxosservice(monmap) restart
  275. 2013-03-22 17:10:41.336344 7ffa31064700 10 mon.1@2(probing).paxosservice(auth) restart
  276. 2013-03-22 17:10:41.336348 7ffa31064700 10 mon.1@2(probing) e4 cancel_probe_timeout (none scheduled)
  277. 2013-03-22 17:10:41.336354 7ffa31064700 10 mon.1@2(probing) e4 reset_probe_timeout 0x18ae6b0 after 2 seconds
  278. 2013-03-22 17:10:41.336400 7ffa31064700 10 mon.1@2(probing) e4 probing other monitors
  279.  
  280.  
  281. 2013-03-22 17:22:42.232040 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- election(3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b propose 269) v3 -- ?+0 0x1e1cd80
  282. 2013-03-22 17:22:42.232092 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.1 172.16.0.3:6789/0 -- election(3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b propose 269) v3 -- ?+0 0x1e1cb40
  283. 2013-03-22 17:22:42.367314 7ffa31064700 11 mon.1@2(electing) e4 tick
  284. 2013-03-22 17:22:42.367427 7ffa31064700 20 mon.1@2(electing) e4 have connection
  285. 2013-03-22 17:22:42.367438 7ffa31064700 1 mon.1@2(electing) e4 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum
  286. 2013-03-22 17:22:42.367451 7ffa31064700 1 -- 172.16.0.4:6789/0 mark_down 0x1f1e6e0 -- pipe dne
  287. 2013-03-22 17:22:45.194143 7ffa2f55e700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x3283780 sd=40 :6789 s=0 pgs=0 cs=0 l=0).accept sd=40 172.16.0.2:39143/0
  288. 2013-03-22 17:22:45.194372 7ffa2f55e700 10 mon.1@2(electing) e4 ms_verify_authorizer 172.16.0.2:0/8285 client protocol 0
  289. 2013-03-22 17:22:45.194776 7ffa30863700 1 -- 172.16.0.4:6789/0 <== client.? 172.16.0.2:0/8285 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x1e21000 con 0x1f1e160
  290. 2013-03-22 17:22:45.194834 7ffa30863700 20 mon.1@2(electing) e4 have connection
  291. 2013-03-22 17:22:45.194844 7ffa30863700 5 mon.1@2(electing) e4 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
  292. 2013-03-22 17:22:46.151633 7ffa2f45d700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x3283500 sd=41 :6789 s=0 pgs=0 cs=0 l=0).accept sd=41 172.16.0.3:60613/0
  293. 2013-03-22 17:22:46.151925 7ffa2f45d700 10 mon.1@2(electing) e4 ms_verify_authorizer 172.16.0.3:0/11161 client protocol 0
  294. 2013-03-22 17:22:46.152470 7ffa30863700 1 -- 172.16.0.4:6789/0 <== client.? 172.16.0.3:0/11161 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x1e21200 con 0x1f1e000
  295. 2013-03-22 17:22:46.152524 7ffa30863700 20 mon.1@2(electing) e4 have connection
  296. 2013-03-22 17:22:46.152534 7ffa30863700 5 mon.1@2(electing) e4 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
  297. 2013-03-22 17:22:47.232197 7ffa31064700 5 mon.1@2(electing).elector(269) election timer expired
  298. 2013-03-22 17:22:47.232226 7ffa31064700 5 mon.1@2(electing).elector(269) start -- can i be leader?
  299. 2013-03-22 17:22:47.232307 7ffa31064700 1 mon.1@2(electing).elector(269) init, last seen epoch 269
  300. 2013-03-22 17:22:47.232329 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.0 172.16.0.2:6789/0 -- election(3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b propose 269) v3 -- ?+0 0x1e1c900
  301. 2013-03-22 17:22:47.232380 7ffa31064700 1 -- 172.16.0.4:6789/0 --> mon.1 172.16.0.3:6789/0 -- election(3d9571c0-b86c-4b6c-85b6-dc0a7aa8923b propose 269) v3 -- ?+0 0x1e1c6c0
  302. 2013-03-22 17:22:47.367565 7ffa31064700 11 mon.1@2(electing) e4 tick
  303. 2013-03-22 17:22:47.367679 7ffa31064700 20 mon.1@2(electing) e4 have connection
  304. 2013-03-22 17:22:47.367687 7ffa31064700 5 mon.1@2(electing) e4 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
  305. 2013-03-22 17:22:47.367701 7ffa31064700 20 mon.1@2(electing) e4 have connection
  306. 2013-03-22 17:22:47.367706 7ffa31064700 5 mon.1@2(electing) e4 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
  307. 2013-03-22 17:22:48.194007 7ffa30863700 10 mon.1@2(electing) e4 ms_handle_reset 0x1f1e160 172.16.0.2:0/8285
  308. 2013-03-22 17:22:49.151608 7ffa30863700 10 mon.1@2(electing) e4 ms_handle_reset 0x1f1e000 172.16.0.3:0/11161
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!