API tools faq
paste
yasu

ceph-mds.2.log

yasu
Sep 19th, 2013
160
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 20.61 KB | None | 0 0
raw download clone embed print report
  1. 2013-09-19 00:22:11.184581 7f05af7fe7c0 0 ceph version 0.67.3 (408cd61584c72c0d
  2. 97b774b3d8f95c6b1b06341a), process ceph-mds, pid 25877
  3. 2013-09-19 00:22:11.243415 7f05ad7f2700 1 mds.-1.0 handle_mds_map standby
  4. 2013-09-19 00:22:11.252925 7f05ad7f2700 1 mds.0.9 handle_mds_map i am now mds.0
  5. .9
  6. 2013-09-19 00:22:11.252931 7f05ad7f2700 1 mds.0.9 handle_mds_map state change u
  7. p:standby --> up:replay
  8. 2013-09-19 00:22:11.252937 7f05ad7f2700 1 mds.0.9 replay_start
  9. 2013-09-19 00:22:11.252943 7f05ad7f2700 1 mds.0.9 recovery set is
  10. 2013-09-19 00:22:11.252953 7f05ad7f2700 1 mds.0.9 need osdmap epoch 14780, hav
  11. e 14779
  12. 2013-09-19 00:22:11.252954 7f05ad7f2700 1 mds.0.9 waiting for osdmap 14780 (wh
  13. ich blacklists prior instance)
  14. 2013-09-19 00:22:11.257172 7f05abfef700 -1 mds/MDSTable.cc: In function 'void MD
  15. STable::load_2(int, ceph::bufferlist&, Context*)' thread 7f05abfef700 time 2013-
  16. 09-19 00:22:11.254947
  17. mds/MDSTable.cc: 152: FAILED assert(r >= 0)
  18.  
  19. ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
  20. 1: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3b6) [0x812f06]
  21. 2: (Context::complete(int)+0xa) [0x6236ca]
  22. 3: (Objecter::check_op_pool_dne(Objecter::Op*)+0x229) [0x8561f9]
  23. 4: (Objecter::C_Op_Map_Latest::finish(int)+0x1cf) [0x85650f]
  24. 5: (Context::complete(int)+0xa) [0x6236ca]
  25. 6: (Finisher::finisher_thread_entry()+0x1c8) [0x9ef588]
  26. 7: /lib64/libpthread.so.0() [0x310ea07d15]
  27. 8: (clone()+0x6d) [0x310def253d]
  28. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to int
  29. erpret this.
  30.  
  31. --- begin dump of recent events ---
  32. -118> 2013-09-19 00:22:11.183054 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  33. and perfcounters_dump hook 0x3bf8010
  34. -117> 2013-09-19 00:22:11.183087 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  35. and 1 hook 0x3bf8010
  36. -116> 2013-09-19 00:22:11.183091 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  37. and perf dump hook 0x3bf8010
  38. -115> 2013-09-19 00:22:11.183096 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  39. and perfcounters_schema hook 0x3bf8010
  40. -114> 2013-09-19 00:22:11.183099 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  41. and 2 hook 0x3bf8010
  42. -113> 2013-09-19 00:22:11.183102 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  43. and perf schema hook 0x3bf8010
  44. -112> 2013-09-19 00:22:11.183106 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  45. and config show hook 0x3bf8010
  46. -111> 2013-09-19 00:22:11.183116 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  47. and config set hook 0x3bf8010
  48. -110> 2013-09-19 00:22:11.183120 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  49. and config get hook 0x3bf8010
  50. -109> 2013-09-19 00:22:11.183122 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  51. and log flush hook 0x3bf8010
  52. -108> 2013-09-19 00:22:11.183125 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  53. and log dump hook 0x3bf8010
  54. -107> 2013-09-19 00:22:11.183128 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  55. and log reopen hook 0x3bf8010
  56. -106> 2013-09-19 00:22:11.184581 7f05af7fe7c0 0 ceph version 0.67.3 (408cd615
  57. 84c72c0d97b774b3d8f95c6b1b06341a), process ceph-mds, pid 25877
  58. -105> 2013-09-19 00:22:11.184764 7f05af7fe7c0 1 accepter.accepter.bind my_ins
  59. t.addr is 0.0.0.0:6801/25877 need_addr=1
  60. -104> 2013-09-19 00:22:11.185073 7f05af7fe7c0 1 finished global_init_daemoniz
  61. e
  62. -103> 2013-09-19 00:22:11.185180 7f05af7fe7c0 5 asok(0x3c30000) init /var/run
  63. /ceph/ceph-mds.2.asok
  64. -102> 2013-09-19 00:22:11.185197 7f05af7fe7c0 5 asok(0x3c30000) bind_and_list
  65. en /var/run/ceph/ceph-mds.2.asok
  66. -101> 2013-09-19 00:22:11.185220 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  67. and 0 hook 0x3bf0038
  68. -100> 2013-09-19 00:22:11.185227 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  69. and version hook 0x3bf0038
  70. -99> 2013-09-19 00:22:11.185231 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  71. and git_version hook 0x3bf0038
  72. -98> 2013-09-19 00:22:11.185233 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  73. and help hook 0x3bf8050
  74. -97> 2013-09-19 00:22:11.185236 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  75. and get_command_descriptions hook 0x3bf8040
  76. -96> 2013-09-19 00:22:11.185262 7f05af7fe7c0 10 monclient(hunting): build_ini
  77. tial_monmap
  78. -95> 2013-09-19 00:22:11.185257 7f05ae7f4700 5 asok(0x3c30000) entry start
  79. -94> 2013-09-19 00:22:11.185328 7f05af7fe7c0 1 -- 0.0.0.0:6801/25877 messeng
  80. er.start
  81. -93> 2013-09-19 00:22:11.185362 7f05af7fe7c0 5 adding auth protocol: none
  82. -92> 2013-09-19 00:22:11.185368 7f05af7fe7c0 5 adding auth protocol: none
  83. -91> 2013-09-19 00:22:11.185656 7f05af7fe7c0 1 accepter.accepter.start
  84. -90> 2013-09-19 00:22:11.185682 7f05af7fe7c0 10 monclient(hunting): init
  85. -89> 2013-09-19 00:22:11.185686 7f05af7fe7c0 5 adding auth protocol: none
  86. -88> 2013-09-19 00:22:11.185691 7f05af7fe7c0 10 monclient(hunting): auth_supp
  87. orted 1 method none
  88. -87> 2013-09-19 00:22:11.185754 7f05af7fe7c0 10 monclient(hunting): _reopen_s
  89. ession rank -1 name
  90. -86> 2013-09-19 00:22:11.185819 7f05af7fe7c0 10 monclient(hunting): picked mo
  91. n.3 con 0x3c69080 addr 128.114.52.69:6789/0
  92. -85> 2013-09-19 00:22:11.185853 7f05af7fe7c0 10 monclient(hunting): _send_mon
  93. _message to mon.3 at 128.114.52.69:6789/0
  94. -84> 2013-09-19 00:22:11.185858 7f05af7fe7c0 1 -- 0.0.0.0:6801/25877 --> 128
  95. .114.52.69:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x3c28000 con 0x3c
  96. 69080
  97. -83> 2013-09-19 00:22:11.185868 7f05af7fe7c0 10 monclient(hunting): renew_sub
  98. s
  99. -82> 2013-09-19 00:22:11.186979 7f05ab7ee700 1 -- 128.114.52.68:6801/25877 l
  100. earned my addr 128.114.52.68:6801/25877
  101. -81> 2013-09-19 00:22:11.188114 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  102. == mon.3 128.114.52.69:6789/0 1 ==== mon_map v1 ==== 755+0+0 (124592334 0 0) 0x3
  103. d08000 con 0x3c69080
  104. -80> 2013-09-19 00:22:11.188159 7f05ad7f2700 10 monclient(hunting): handle_mo
  105. nmap mon_map v1
  106. -79> 2013-09-19 00:22:11.188180 7f05ad7f2700 10 monclient(hunting): got monm
  107. ap 1, mon.3 is now rank 3
  108. -78> 2013-09-19 00:22:11.188185 7f05ad7f2700 10 monclient(hunting): dump:
  109. epoch 1
  110. fsid 6ef8e7bb-745c-4e20-8946-8808b9843380
  111. last_changed 0.000000
  112. created 0.000000
  113. 0: 128.114.52.59:6789/0 mon.0
  114. 1: 128.114.52.67:6789/0 mon.1
  115. 2: 128.114.52.68:6789/0 mon.2
  116. 3: 128.114.52.69:6789/0 mon.3
  117. 4: 128.114.52.70:6789/0 mon.4
  118.  
  119. -77> 2013-09-19 00:22:11.188248 7f05ad7f2700 1 monclient(hunting): found mon
  120. .3
  121. -76> 2013-09-19 00:22:11.188258 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  122. == mon.3 128.114.52.69:6789/0 2 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+
  123. 0 (483986750 0 0) 0x3c50600 con 0x3c69080
  124. -75> 2013-09-19 00:22:11.188290 7f05ad7f2700 10 monclient: my global_id is 51
  125. 48
  126. -74> 2013-09-19 00:22:11.188295 7f05ad7f2700 10 monclient: _send_mon_message
  127. to mon.3 at 128.114.52.69:6789/0
  128. -73> 2013-09-19 00:22:11.188316 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  129. -> 128.114.52.69:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x3cc81c0 con 0x
  130. 3c69080
  131. -72> 2013-09-19 00:22:11.188347 7f05ad7f2700 10 monclient: _check_auth_rotati
  132. ng renewing rotating keys (they expired before 2013-09-19 00:21:41.188347)
  133. -71> 2013-09-19 00:22:11.188366 7f05af7fe7c0 5 monclient: authenticate succe
  134. ss, global_id 5148
  135. -70> 2013-09-19 00:22:11.188378 7f05af7fe7c0 5 asok(0x3c30000) register_comm
  136. and objecter_requests hook 0x3bf8080
  137. -69> 2013-09-19 00:22:11.188412 7f05af7fe7c0 10 monclient: _send_mon_message
  138. to mon.3 at 128.114.52.69:6789/0
  139. -68> 2013-09-19 00:22:11.188418 7f05af7fe7c0 1 -- 128.114.52.68:6801/25877 -
  140. -> 128.114.52.69:6789/0 -- mdsbeacon(5148/2 up:boot seq 1 v0) v2 -- ?+0 0x3ce858
  141. 0 con 0x3c69080
  142. -67> 2013-09-19 00:22:11.188441 7f05af7fe7c0 10 monclient: renew_subs
  143. -66> 2013-09-19 00:22:11.188446 7f05af7fe7c0 10 monclient: _send_mon_message
  144. to mon.3 at 128.114.52.69:6789/0
  145. -65> 2013-09-19 00:22:11.188451 7f05af7fe7c0 1 -- 128.114.52.68:6801/25877 -
  146. -> 128.114.52.69:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x3cc88
  147. c0 con 0x3c69080
  148. -64> 2013-09-19 00:22:11.188467 7f05af7fe7c0 10 monclient: renew_subs
  149. -63> 2013-09-19 00:22:11.188472 7f05af7fe7c0 10 monclient: _send_mon_message
  150. to mon.3 at 128.114.52.69:6789/0
  151. -62> 2013-09-19 00:22:11.188477 7f05af7fe7c0 1 -- 128.114.52.68:6801/25877 -
  152. -> 128.114.52.69:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?
  153. +0 0x3cc8700 con 0x3c69080
  154. -61> 2013-09-19 00:22:11.189062 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  155. == mon.3 128.114.52.69:6789/0 3 ==== mon_map v1 ==== 755+0+0 (124592334 0 0) 0x3
  156. d083c0 con 0x3c69080
  157. -60> 2013-09-19 00:22:11.189096 7f05ad7f2700 10 monclient: handle_monmap mon_
  158. map v1
  159. -59> 2013-09-19 00:22:11.189107 7f05ad7f2700 10 monclient: got monmap 1, mon
  160. .3 is now rank 3
  161. -58> 2013-09-19 00:22:11.189110 7f05ad7f2700 10 monclient: dump:
  162. epoch 1
  163. fsid 6ef8e7bb-745c-4e20-8946-8808b9843380
  164. last_changed 0.000000
  165. created 0.000000
  166. 0: 128.114.52.59:6789/0 mon.0
  167. 1: 128.114.52.67:6789/0 mon.1
  168. 2: 128.114.52.68:6789/0 mon.2
  169. 3: 128.114.52.69:6789/0 mon.3
  170. 4: 128.114.52.70:6789/0 mon.4
  171.  
  172. -57> 2013-09-19 00:22:11.189133 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  173. == mon.3 128.114.52.69:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (242
  174. 0943214 0 0) 0x3cc81c0 con 0x3c69080
  175. -56> 2013-09-19 00:22:11.189144 7f05ad7f2700 10 monclient: handle_subscribe_a
  176. ck sent 2013-09-19 00:22:11.185874 renew after 2013-09-19 00:24:41.185874
  177. -55> 2013-09-19 00:22:11.193262 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  178. == mon.3 128.114.52.69:6789/0 5 ==== osd_map(14779..14779 src has 1..14779) v3 =
  179. === 65380+0+0 (2114969669 0 0) 0x3c28480 con 0x3c69080
  180. -54> 2013-09-19 00:22:11.194440 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  181. == mon.3 128.114.52.69:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (242
  182. 0943214 0 0) 0x3cc8a80 con 0x3c69080
  183. -53> 2013-09-19 00:22:11.194457 7f05ad7f2700 10 monclient: handle_subscribe_a
  184. ck sent 0.000000, ignoring
  185. -52> 2013-09-19 00:22:11.194467 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  186. == mon.3 128.114.52.69:6789/0 7 ==== mdsmap(e 3458) v1 ==== 583+0+0 (4114617233
  187. 0 0) 0x3c50a00 con 0x3c69080
  188. -51> 2013-09-19 00:22:11.195370 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  189. == mon.3 128.114.52.69:6789/0 8 ==== osd_map(14779..14779 src has 1..14779) v3 =
  190. === 65380+0+0 (2114969669 0 0) 0x3c28900 con 0x3c69080
  191. -50> 2013-09-19 00:22:11.195399 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  192. == mon.3 128.114.52.69:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (242
  193. 0943214 0 0) 0x3cc88c0 con 0x3c69080
  194. -49> 2013-09-19 00:22:11.195409 7f05ad7f2700 10 monclient: handle_subscribe_a
  195. ck sent 0.000000, ignoring
  196. -48> 2013-09-19 00:22:11.243361 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  197. == mon.3 128.114.52.69:6789/0 10 ==== mdsmap(e 3459) v1 ==== 575+0+0 (2995627428
  198. 0 0) 0x3c50800 con 0x3c69080
  199. -47> 2013-09-19 00:22:11.243403 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 m
  200. ark_down 128.114.52.68:6804/6062 -- pipe dne
  201. -46> 2013-09-19 00:22:11.243415 7f05ad7f2700 1 mds.-1.0 handle_mds_map stand
  202. by
  203. -45> 2013-09-19 00:22:11.252878 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  204. == mon.3 128.114.52.69:6789/0 11 ==== mdsmap(e 3460) v1 ==== 583+0+0 (1534222726
  205. 0 0) 0x3c50c00 con 0x3c69080
  206. -44> 2013-09-19 00:22:11.252925 7f05ad7f2700 1 mds.0.9 handle_mds_map i am n
  207. ow mds.0.9
  208. -43> 2013-09-19 00:22:11.252931 7f05ad7f2700 1 mds.0.9 handle_mds_map state
  209. change up:standby --> up:replay
  210. -42> 2013-09-19 00:22:11.252937 7f05ad7f2700 1 mds.0.9 replay_start
  211. -41> 2013-09-19 00:22:11.252943 7f05ad7f2700 1 mds.0.9 recovery set is
  212. -40> 2013-09-19 00:22:11.252953 7f05ad7f2700 1 mds.0.9 need osdmap epoch 14
  213. 780, have 14779
  214. -39> 2013-09-19 00:22:11.252954 7f05ad7f2700 1 mds.0.9 waiting for osdmap 1
  215. 4780 (which blacklists prior instance)
  216. -38> 2013-09-19 00:22:11.252963 7f05ad7f2700 10 monclient: renew_subs
  217. -37> 2013-09-19 00:22:11.252967 7f05ad7f2700 10 monclient: _send_mon_message
  218. to mon.3 at 128.114.52.69:6789/0
  219. -36> 2013-09-19 00:22:11.252972 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  220. -> 128.114.52.69:6789/0 -- mon_subscribe({mdsmap=3461+,monmap=2+,osdmap=14780})
  221. v2 -- ?+0 0x3cc88c0 con 0x3c69080
  222. -35> 2013-09-19 00:22:11.253848 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  223. == mon.3 128.114.52.69:6789/0 12 ==== osd_map(14780..14780 src has 1..14780) v3
  224. ==== 312+0+0 (3860788984 0 0) 0x3c28900 con 0x3c69080
  225. -34> 2013-09-19 00:22:11.253902 7f05ad7f2700 10 monclient: renew_subs
  226. -33> 2013-09-19 00:22:11.253916 7f05ad7f2700 10 monclient: _send_mon_message
  227. to mon.3 at 128.114.52.69:6789/0
  228. -32> 2013-09-19 00:22:11.253923 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  229. -> 128.114.52.69:6789/0 -- mon_subscribe({mdsmap=3461+,monmap=2+,osdmap=14781})
  230. v2 -- ?+0 0x3cc8fc0 con 0x3c69080
  231. -31> 2013-09-19 00:22:11.253935 7f05ad7f2700 10 monclient: get_version osdmap
  232. req 0x3bea440
  233. -30> 2013-09-19 00:22:11.253939 7f05ad7f2700 10 monclient: _send_mon_message
  234. to mon.3 at 128.114.52.69:6789/0
  235. -29> 2013-09-19 00:22:11.253943 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  236. -> 128.114.52.69:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x3cd
  237. 84e0 con 0x3c69080
  238. -28> 2013-09-19 00:22:11.253959 7f05ad7f2700 10 monclient: get_version osdmap
  239. req 0x3bea3a0
  240. -27> 2013-09-19 00:22:11.253962 7f05ad7f2700 10 monclient: _send_mon_message
  241. to mon.3 at 128.114.52.69:6789/0
  242. -26> 2013-09-19 00:22:11.253966 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  243. -> 128.114.52.69:6789/0 -- mon_get_version(what=osdmap handle=2) v1 -- ?+0 0x3cd
  244. 8820 con 0x3c69080
  245. -25> 2013-09-19 00:22:11.253981 7f05ad7f2700 10 monclient: get_version osdmap
  246. req 0x3be93a0
  247. -24> 2013-09-19 00:22:11.253984 7f05ad7f2700 10 monclient: _send_mon_message
  248. to mon.3 at 128.114.52.69:6789/0
  249. -23> 2013-09-19 00:22:11.253988 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  250. -> 128.114.52.69:6789/0 -- mon_get_version(what=osdmap handle=3) v1 -- ?+0 0x3cd
  251. 8680 con 0x3c69080
  252. -22> 2013-09-19 00:22:11.254002 7f05ad7f2700 10 monclient: get_version osdmap
  253. req 0x3be9260
  254. -21> 2013-09-19 00:22:11.254005 7f05ad7f2700 10 monclient: _send_mon_message
  255. to mon.3 at 128.114.52.69:6789/0
  256. -20> 2013-09-19 00:22:11.254009 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  257. -> 128.114.52.69:6789/0 -- mon_get_version(what=osdmap handle=4) v1 -- ?+0 0x3cd
  258. 89c0 con 0x3c69080
  259. -19> 2013-09-19 00:22:11.254031 7f05ad7f2700 1 mds.0.journaler(ro) recover s
  260. tart
  261. -18> 2013-09-19 00:22:11.254035 7f05ad7f2700 1 mds.0.journaler(ro) read_head
  262.  
  263. -17> 2013-09-19 00:22:11.254052 7f05ad7f2700 10 monclient: get_version osdmap
  264. req 0x3bea640
  265. -16> 2013-09-19 00:22:11.254055 7f05ad7f2700 10 monclient: _send_mon_message
  266. to mon.3 at 128.114.52.69:6789/0
  267. -15> 2013-09-19 00:22:11.254060 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 -
  268. -> 128.114.52.69:6789/0 -- mon_get_version(what=osdmap handle=5) v1 -- ?+0 0x3cd
  269. 8820 con 0x3c69080
  270. -14> 2013-09-19 00:22:11.254078 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  271. == mon.3 128.114.52.69:6789/0 13 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (24
  272. 20943214 0 0) 0x3cc8c40 con 0x3c69080
  273. -13> 2013-09-19 00:22:11.254085 7f05ad7f2700 10 monclient: handle_subscribe_a
  274. ck sent 2013-09-19 00:22:11.252965 renew after 2013-09-19 00:24:41.252965
  275. -12> 2013-09-19 00:22:11.254879 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  276. == mon.3 128.114.52.69:6789/0 14 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (24
  277. 20943214 0 0) 0x3cc8a80 con 0x3c69080
  278. -11> 2013-09-19 00:22:11.254890 7f05ad7f2700 10 monclient: handle_subscribe_a
  279. ck sent 0.000000, ignoring
  280. -10> 2013-09-19 00:22:11.254897 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  281. == mon.3 128.114.52.69:6789/0 15 ==== mon_check_map_ack(handle=1 version=14780)
  282. v2 ==== 24+0+0 (3482297372 0 0) 0x3cc81c0 con 0x3c69080
  283. -9> 2013-09-19 00:22:11.254906 7f05ad7f2700 10 monclient: handle_get_version
  284. _reply finishing 0x3bea440 version 14780
  285. -8> 2013-09-19 00:22:11.254917 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  286. == mon.3 128.114.52.69:6789/0 16 ==== mon_check_map_ack(handle=2 version=14780)
  287. v2 ==== 24+0+0 (78878079 0 0) 0x3cc9180 con 0x3c69080
  288. -7> 2013-09-19 00:22:11.254925 7f05ad7f2700 10 monclient: handle_get_version
  289. _reply finishing 0x3bea3a0 version 14780
  290. -6> 2013-09-19 00:22:11.255351 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  291. == mon.3 128.114.52.69:6789/0 17 ==== mon_check_map_ack(handle=3 version=14780)
  292. v2 ==== 24+0+0 (3204208625 0 0) 0x3cc8fc0 con 0x3c69080
  293. -5> 2013-09-19 00:22:11.255362 7f05ad7f2700 10 monclient: handle_get_version
  294. _reply finishing 0x3be93a0 version 14780
  295. -4> 2013-09-19 00:22:11.255376 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  296. == mon.3 128.114.52.69:6789/0 18 ==== mon_check_map_ack(handle=4 version=14780)
  297. v2 ==== 24+0+0 (2535948616 0 0) 0x3cc88c0 con 0x3c69080
  298. -3> 2013-09-19 00:22:11.255393 7f05ad7f2700 10 monclient: handle_get_version
  299. _reply finishing 0x3be9260 version 14780
  300. -2> 2013-09-19 00:22:11.255404 7f05ad7f2700 1 -- 128.114.52.68:6801/25877 <
  301. == mon.3 128.114.52.69:6789/0 19 ==== mon_check_map_ack(handle=5 version=14780)
  302. v2 ==== 24+0+0 (761807814 0 0) 0x3cc8700 con 0x3c69080
  303. -1> 2013-09-19 00:22:11.255410 7f05ad7f2700 10 monclient: handle_get_version
  304. _reply finishing 0x3bea640 version 14780
  305. 0> 2013-09-19 00:22:11.257172 7f05abfef700 -1 mds/MDSTable.cc: In function
  306. 'void MDSTable::load_2(int, ceph::bufferlist&, Context*)' thread 7f05abfef700 ti
  307. me 2013-09-19 00:22:11.254947
  308. mds/MDSTable.cc: 152: FAILED assert(r >= 0)
  309.  
  310. ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
  311. 1: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3b6) [0x812f06]
  312. 2: (Context::complete(int)+0xa) [0x6236ca]
  313. 3: (Objecter::check_op_pool_dne(Objecter::Op*)+0x229) [0x8561f9]
  314. 4: (Objecter::C_Op_Map_Latest::finish(int)+0x1cf) [0x85650f]
  315. 5: (Context::complete(int)+0xa) [0x6236ca]
  316. 6: (Finisher::finisher_thread_entry()+0x1c8) [0x9ef588]
  317. 7: /lib64/libpthread.so.0() [0x310ea07d15]
  318. 8: (clone()+0x6d) [0x310def253d]
  319. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to int
  320. erpret this.
  321.  
  322. --- logging levels ---
  323. 0/ 5 none
  324. 0/ 1 lockdep
  325. 0/ 1 context
  326. 1/ 1 crush
  327. 1/ 1 mds
  328. 1/ 5 mds_balancer
  329. 1/ 5 mds_locker
  330. 1/ 5 mds_log
  331. 1/ 5 mds_log_expire
  332. 1/ 5 mds_migrator
  333. 0/ 1 buffer
  334. 0/ 1 timer
  335. 0/ 1 filer
  336. 0/ 1 striper
  337. 0/ 1 objecter
  338. 0/ 5 rados
  339. 0/ 5 rbd
  340. 0/ 5 journaler
  341. 0/ 5 objectcacher
  342. 0/ 5 client
  343. 0/ 5 osd
  344. 0/ 5 optracker
  345. 0/ 5 objclass
  346. 1/ 3 filestore
  347. 1/ 3 journal
  348. 0/ 5 ms
  349. 1/ 5 mon
  350. 0/10 monc
  351. 1/ 5 paxos
  352. 0/ 5 tp
  353. 1/ 5 auth
  354. 1/ 5 crypto
  355. 1/ 1 finisher
  356. 1/ 5 heartbeatmap
  357. 1/ 5 perfcounter
  358. 1/ 5 rgw
  359. 1/ 5 hadoop
  360. 1/ 5 javaclient
  361. 1/ 5 asok
  362. 1/ 1 throttle
  363. -2/-2 (syslog threshold)
  364. -1/-1 (stderr threshold)
  365. max_recent 10000
  366. max_new 1000
  367. log_file /ceph/var/log/ceph/ceph-mds.2.log
  368. --- end dump of recent events ---
  369. 2013-09-19 00:22:11.259934 7f05abfef700 -1 *** Caught signal (Aborted) **
  370. in thread 7f05abfef700
  371.  
  372. ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
  373. 1: /ceph/bin/ceph-mds() [0xa19ae2]
  374. 2: /lib64/libpthread.so.0() [0x310ea0f000]
  375. 3: (gsignal()+0x35) [0x310de35c55]
  376. 4: (abort()+0x148) [0x310de37408]
  377. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x3110260e1d]
  378. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to int
  379. erpret this.
  380.  
  381. --- begin dump of recent events ---
  382. 0> 2013-09-19 00:22:11.259934 7f05abfef700 -1 *** Caught signal (Aborted) *
  383. *
  384. in thread 7f05abfef700
  385.  
  386. ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
  387. 1: /ceph/bin/ceph-mds() [0xa19ae2]
  388. 2: /lib64/libpthread.so.0() [0x310ea0f000]
  389. 3: (gsignal()+0x35) [0x310de35c55]
  390. 4: (abort()+0x148) [0x310de37408]
  391. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x3110260e1d]
  392. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to int
  393. erpret this.
  394.  
  395. --- logging levels ---
  396. 0/ 5 none
  397. 0/ 1 lockdep
  398. 0/ 1 context
  399. 1/ 1 crush
  400. 1/ 1 mds
  401. 1/ 5 mds_balancer
  402. 1/ 5 mds_locker
  403. 1/ 5 mds_log
  404. 1/ 5 mds_log_expire
  405. 1/ 5 mds_migrator
  406. 0/ 1 buffer
  407. 0/ 1 timer
  408. 0/ 1 filer
  409. 0/ 1 striper
  410. 0/ 1 objecter
  411. 0/ 5 rados
  412. 0/ 5 rbd
  413. 0/ 5 journaler
  414. 0/ 5 objectcacher
  415. 0/ 5 client
  416. 0/ 5 osd
  417. 0/ 5 optracker
  418. 0/ 5 objclass
  419. 1/ 3 filestore
  420. 1/ 3 journal
  421. 0/ 5 ms
  422. 1/ 5 mon
  423. 0/10 monc
  424. 1/ 5 paxos
  425. 0/ 5 tp
  426. 1/ 5 auth
  427. 1/ 5 crypto
  428. 1/ 1 finisher
  429. 1/ 5 heartbeatmap
  430. 1/ 5 perfcounter
  431. 1/ 5 rgw
  432. 1/ 5 hadoop
  433. 1/ 5 javaclient
  434. 1/ 5 asok
  435. 1/ 1 throttle
  436. -2/-2 (syslog threshold)
  437. -1/-1 (stderr threshold)
  438. max_recent 10000
  439. max_new 1000
  440. log_file /ceph/var/log/ceph/ceph-mds.2.log
  441. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!