API tools faq
paste
Advertisement
Guest User

gippa-mds

a guest
May 23rd, 2013
332
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 14.67 KB | None | 0 0
raw download clone embed print report
  1. created 2013-05-23 22:48:48.515786
  2. 0: 10.123.200.188:6789/0 mon.stg01
  3. 1: 10.123.200.189:6789/0 mon.stg02
  4.  
  5. -69> 2013-05-23 23:00:33.261036 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (921860333 0 0) 0x29d2540 con 0x29c4580
  6. -68> 2013-05-23 23:00:33.261071 7fa39e28e700 10 monclient: handle_subscribe_ack sent 2013-05-23 23:00:33.242872 renew after 2013-05-23 23:03:03.242872
  7. -67> 2013-05-23 23:00:33.261084 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1908677215 0 0) 0x29afa00 con 0x29c4580
  8. -66> 2013-05-23 23:00:33.261212 7fa39e28e700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-05-23 23:00:03.261211)
  9. -65> 2013-05-23 23:00:33.261237 7fa3a379b780 10 monclient: wait_auth_rotating done
  10. -64> 2013-05-23 23:00:33.261287 7fa3a379b780 5 asok(0x29a2000) register_command objecter_requests hook 0x298a260
  11. -63> 2013-05-23 23:00:33.261390 7fa3a379b780 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  12. -62> 2013-05-23 23:00:33.261403 7fa3a379b780 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mdsbeacon(4516/stg02 up:boot seq 1 v0) v2 -- ?+0 0x29dc2c0 con 0x29c4580
  13. -61> 2013-05-23 23:00:33.261442 7fa3a379b780 10 monclient: renew_subs
  14. -60> 2013-05-23 23:00:33.261446 7fa3a379b780 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  15. -59> 2013-05-23 23:00:33.261450 7fa3a379b780 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x29d21c0 con 0x29c4580
  16. -58> 2013-05-23 23:00:33.261462 7fa3a379b780 10 monclient: renew_subs
  17. -57> 2013-05-23 23:00:33.261464 7fa3a379b780 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  18. -56> 2013-05-23 23:00:33.261468 7fa3a379b780 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x29d2a80 con 0x29c4580
  19. -55> 2013-05-23 23:00:33.299133 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 8 ==== osd_map(16..16 src has 1..16) v3 ==== 2573+0+0 (1805929700 0 0) 0x29a0b40 con 0x29c4580
  20. -54> 2013-05-23 23:00:33.299400 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (921860333 0 0) 0x29d21c0 con 0x29c4580
  21. -53> 2013-05-23 23:00:33.299417 7fa39e28e700 10 monclient: handle_subscribe_ack sent 2013-05-23 23:00:33.261444 renew after 2013-05-23 23:03:03.261444
  22. -52> 2013-05-23 23:00:33.299435 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 10 ==== mdsmap(e 11) v1 ==== 587+0+0 (3695662834 0 0) 0x29af800 con 0x29c4580
  23. -51> 2013-05-23 23:00:33.299449 7fa39e28e700 5 mds.-1.0 handle_mds_map epoch 11 from mon.0
  24. -50> 2013-05-23 23:00:33.299542 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 11 ==== osd_map(16..16 src has 1..16) v3 ==== 2573+0+0 (1805929700 0 0) 0x29a0fc0 con 0x29c4580
  25. -49> 2013-05-23 23:00:33.299567 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (921860333 0 0) 0x29d2380 con 0x29c4580
  26. -48> 2013-05-23 23:00:33.299576 7fa39e28e700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  27. -47> 2013-05-23 23:00:33.370373 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 13 ==== mdsmap(e 12) v1 ==== 794+0+0 (932971675 0 0) 0x29af400 con 0x29c4580
  28. -46> 2013-05-23 23:00:33.370433 7fa39e28e700 5 mds.-1.-1 handle_mds_map epoch 12 from mon.0
  29. -45> 2013-05-23 23:00:33.370473 7fa39e28e700 1 mds.-1.0 handle_mds_map standby
  30. -44> 2013-05-23 23:00:36.242858 7fa39d28c700 10 monclient: tick
  31. -43> 2013-05-23 23:00:36.242908 7fa39d28c700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-05-23 23:00:06.242905)
  32. -42> 2013-05-23 23:00:36.242927 7fa39d28c700 10 monclient: renew subs? (now: 2013-05-23 23:00:36.242927; renew after: 2013-05-23 23:03:03.261444) -- no
  33. -41> 2013-05-23 23:00:37.261716 7fa39c088700 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  34. -40> 2013-05-23 23:00:37.261766 7fa39c088700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mdsbeacon(4516/stg02 up:standby seq 2 v12) v2 -- ?+0 0x29dc840 con 0x29c4580
  35. -39> 2013-05-23 23:00:37.271339 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 14 ==== mdsbeacon(4516/stg02 up:standby seq 2 v12) v2 ==== 107+0+0 (1058662658 0 0) 0x29dc2c0 con 0x29c4580
  36. -38> 2013-05-23 23:00:40.911514 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 15 ==== mdsmap(e 13) v1 ==== 587+0+0 (406889085 0 0) 0x29afc00 con 0x29c4580
  37. -37> 2013-05-23 23:00:40.911607 7fa39e28e700 5 mds.-1.0 handle_mds_map epoch 13 from mon.0
  38. -36> 2013-05-23 23:00:40.911649 7fa39e28e700 1 -- 10.123.200.189:6800/28919 mark_down 10.123.200.188:6800/27373 -- pipe dne
  39. -35> 2013-05-23 23:00:40.911659 7fa39e28e700 1 mds.0.4 handle_mds_map i am now mds.0.4
  40. -34> 2013-05-23 23:00:40.911663 7fa39e28e700 1 mds.0.4 handle_mds_map state change up:standby --> up:replay
  41. -33> 2013-05-23 23:00:40.911698 7fa39e28e700 1 mds.0.4 replay_start
  42. -32> 2013-05-23 23:00:40.911714 7fa39e28e700 1 mds.0.4 recovery set is
  43. -31> 2013-05-23 23:00:40.911717 7fa39e28e700 1 mds.0.4 need osdmap epoch 19, have 16
  44. -30> 2013-05-23 23:00:40.911719 7fa39e28e700 1 mds.0.4 waiting for osdmap 19 (which blacklists prior instance)
  45. -29> 2013-05-23 23:00:40.911738 7fa39e28e700 10 monclient: renew_subs
  46. -28> 2013-05-23 23:00:40.911744 7fa39e28e700 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  47. -27> 2013-05-23 23:00:40.911751 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mon_subscribe({mdsmap=14+,monmap=2+,osdmap=17}) v2 -- ?+0 0x29d2380 con 0x29c4580
  48. -26> 2013-05-23 23:00:40.911770 7fa39e28e700 1 -- 10.123.200.189:6800/28919 mark_down 10.123.200.188:6800/27373 -- pipe dne
  49. -25> 2013-05-23 23:00:40.911780 7fa39e28e700 5 mds.0.4 handle_mds_failure for myself; not doing anything
  50. -24> 2013-05-23 23:00:40.980826 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 16 ==== osd_map(17..19 src has 1..19) v3 ==== 1046+0+0 (164532596 0 0) 0x29a0d80 con 0x29c4580
  51. -23> 2013-05-23 23:00:40.980964 7fa39e28e700 2 mds.0.4 boot_start 1: opening inotable
  52. -22> 2013-05-23 23:00:40.981173 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6802/27665 -- osd_op(mds.0.4:1 mds0_inotable [read 0~0] 1.b852b893 e19) v4 -- ?+0 0x29a1200 con 0x29c4b00
  53. -21> 2013-05-23 23:00:40.981209 7fa39e28e700 2 mds.0.4 boot_start 1: opening sessionmap
  54. -20> 2013-05-23 23:00:40.981312 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.189:6801/29220 -- osd_op(mds.0.4:2 mds0_sessionmap [read 0~0] 1.3270c60b e19) v4 -- ?+0 0x29a0fc0 con 0x29c4840
  55. -19> 2013-05-23 23:00:40.981334 7fa39e28e700 2 mds.0.4 boot_start 1: opening anchor table
  56. -18> 2013-05-23 23:00:40.981349 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.189:6801/29220 -- osd_op(mds.0.4:3 mds_anchortable [read 0~0] 1.a977f6a7 e19) v4 -- ?+0 0x29a0b40 con 0x29c4840
  57. -17> 2013-05-23 23:00:40.981360 7fa39e28e700 2 mds.0.4 boot_start 1: opening snap table
  58. -16> 2013-05-23 23:00:40.981373 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.189:6801/29220 -- osd_op(mds.0.4:4 mds_snaptable [read 0~0] 1.d90270ad e19) v4 -- ?+0 0x29a0900 con 0x29c4840
  59. -15> 2013-05-23 23:00:40.981382 7fa39e28e700 2 mds.0.4 boot_start 1: opening mds log
  60. -14> 2013-05-23 23:00:40.981390 7fa39e28e700 5 mds.0.log open discovering log bounds
  61. -13> 2013-05-23 23:00:40.981397 7fa39e28e700 1 mds.0.journaler(ro) recover start
  62. -12> 2013-05-23 23:00:40.981400 7fa39e28e700 1 mds.0.journaler(ro) read_head
  63. -11> 2013-05-23 23:00:40.981427 7fa39e28e700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6802/27665 -- osd_op(mds.0.4:5 200.00000000 [read 0~0] 1.844f3494 e19) v4 -- ?+0 0x29a0240 con 0x29c4b00
  64. -10> 2013-05-23 23:00:40.981464 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 17 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (921860333 0 0) 0x29d2c40 con 0x29c4580
  65. -9> 2013-05-23 23:00:40.981476 7fa39e28e700 10 monclient: handle_subscribe_ack sent 2013-05-23 23:00:40.911740 renew after 2013-05-23 23:03:10.911740
  66. -8> 2013-05-23 23:00:40.985156 7fa39e28e700 5 mds.0.4 ms_handle_connect on 10.123.200.189:6801/29220
  67. -7> 2013-05-23 23:00:40.986310 7fa39e28e700 5 mds.0.4 ms_handle_connect on 10.123.200.188:6802/27665
  68. -6> 2013-05-23 23:00:41.262087 7fa39c088700 10 monclient: _send_mon_message to mon.stg01 at 10.123.200.188:6789/0
  69. -5> 2013-05-23 23:00:41.262131 7fa39c088700 1 -- 10.123.200.189:6800/28919 --> 10.123.200.188:6789/0 -- mdsbeacon(4516/stg02 up:replay seq 3 v13) v2 -- ?+0 0x29dcb00 con 0x29c4580
  70. -4> 2013-05-23 23:00:41.264249 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== mon.0 10.123.200.188:6789/0 18 ==== mdsbeacon(4516/stg02 up:replay seq 3 v13) v2 ==== 107+0+0 (4137050802 0 0) 0x29dd080 con 0x29c4580
  71. -3> 2013-05-23 23:00:42.957679 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== osd.0 10.123.200.188:6802/27665 1 ==== osd_op_reply(5 200.00000000 [read 0~0] ack = -2 (No such file or directory)) v4 ==== 111+0+0 (2261481792 0 0) 0x29afe00 con 0x29c4b00
  72. -2> 2013-05-23 23:00:42.957780 7fa39e28e700 0 mds.0.journaler(ro) error getting journal off disk
  73. -1> 2013-05-23 23:00:42.960974 7fa39e28e700 1 -- 10.123.200.189:6800/28919 <== osd.0 10.123.200.188:6802/27665 2 ==== osd_op_reply(1 mds0_inotable [read 0~0] ack = -2 (No such file or directory)) v4 ==== 112+0+0 (1612134461 0 0) 0x2a1c200 con 0x29c4b00
  74. 0> 2013-05-23 23:00:42.963326 7fa39e28e700 -1 mds/MDSTable.cc: In function 'void MDSTable::load_2(int, ceph::bufferlist&, Context*)' thread 7fa39e28e700 time 2013-05-23 23:00:42.961076
  75. mds/MDSTable.cc: 150: FAILED assert(0)
  76.  
  77. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  78. 1: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3bb) [0x6dd2db]
  79. 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe1b) [0x7275bb]
  80. 3: (MDS::handle_core_message(Message*)+0xae7) [0x513c57]
  81. 4: (MDS::_dispatch(Message*)+0x33) [0x513d53]
  82. 5: (MDS::ms_dispatch(Message*)+0xab) [0x515b3b]
  83. 6: (DispatchQueue::entry()+0x393) [0x847ca3]
  84. 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7caeed]
  85. 8: (()+0x6b50) [0x7fa3a3376b50]
  86. 9: (clone()+0x6d) [0x7fa3a1d24a7d]
  87. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  88.  
  89. --- logging levels ---
  90. 0/ 5 none
  91. 0/ 1 lockdep
  92. 0/ 1 context
  93. 1/ 1 crush
  94. 1/ 5 mds
  95. 1/ 5 mds_balancer
  96. 1/ 5 mds_locker
  97. 1/ 5 mds_log
  98. 1/ 5 mds_log_expire
  99. 1/ 5 mds_migrator
  100. 0/ 1 buffer
  101. 0/ 1 timer
  102. 0/ 1 filer
  103. 0/ 1 striper
  104. 0/ 1 objecter
  105. 0/ 5 rados
  106. 0/ 5 rbd
  107. 0/ 5 journaler
  108. 0/ 5 objectcacher
  109. 0/ 5 client
  110. 0/ 5 osd
  111. 0/ 5 optracker
  112. 0/ 5 objclass
  113. 1/ 3 filestore
  114. 1/ 3 journal
  115. 0/ 5 ms
  116. 1/ 5 mon
  117. 0/10 monc
  118. 0/ 5 paxos
  119. 0/ 5 tp
  120. 1/ 5 auth
  121. 1/ 5 crypto
  122. 1/ 1 finisher
  123. 1/ 5 heartbeatmap
  124. 1/ 5 perfcounter
  125. 1/ 5 rgw
  126. 1/ 5 hadoop
  127. 1/ 5 javaclient
  128. 1/ 5 asok
  129. 1/ 1 throttle
  130. -2/-2 (syslog threshold)
  131. -1/-1 (stderr threshold)
  132. max_recent 10000
  133. max_new 1000
  134. log_file /var/log/ceph/ceph-mds.stg02.log
  135. --- end dump of recent events ---
  136. 2013-05-23 23:00:42.968133 7fa39e28e700 -1 *** Caught signal (Aborted) **
  137. in thread 7fa39e28e700
  138.  
  139. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  140. 1: /usr/bin/ceph-mds() [0x874632]
  141. 2: (()+0xf030) [0x7fa3a337f030]
  142. 3: (gsignal()+0x35) [0x7fa3a1c7c475]
  143. 4: (abort()+0x180) [0x7fa3a1c7f6f0]
  144. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa3a24d189d]
  145. 6: (()+0x63996) [0x7fa3a24cf996]
  146. 7: (()+0x639c3) [0x7fa3a24cf9c3]
  147. 8: (()+0x63bee) [0x7fa3a24cfbee]
  148. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x7db0fa]
  149. 10: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3bb) [0x6dd2db]
  150. 11: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe1b) [0x7275bb]
  151. 12: (MDS::handle_core_message(Message*)+0xae7) [0x513c57]
  152. 13: (MDS::_dispatch(Message*)+0x33) [0x513d53]
  153. 14: (MDS::ms_dispatch(Message*)+0xab) [0x515b3b]
  154. 15: (DispatchQueue::entry()+0x393) [0x847ca3]
  155. 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7caeed]
  156. 17: (()+0x6b50) [0x7fa3a3376b50]
  157. 18: (clone()+0x6d) [0x7fa3a1d24a7d]
  158. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  159.  
  160. --- begin dump of recent events ---
  161. 0> 2013-05-23 23:00:42.968133 7fa39e28e700 -1 *** Caught signal (Aborted) **
  162. in thread 7fa39e28e700
  163.  
  164. ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
  165. 1: /usr/bin/ceph-mds() [0x874632]
  166. 2: (()+0xf030) [0x7fa3a337f030]
  167. 3: (gsignal()+0x35) [0x7fa3a1c7c475]
  168. 4: (abort()+0x180) [0x7fa3a1c7f6f0]
  169. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa3a24d189d]
  170. 6: (()+0x63996) [0x7fa3a24cf996]
  171. 7: (()+0x639c3) [0x7fa3a24cf9c3]
  172. 8: (()+0x63bee) [0x7fa3a24cfbee]
  173. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x7db0fa]
  174. 10: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3bb) [0x6dd2db]
  175. 11: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe1b) [0x7275bb]
  176. 12: (MDS::handle_core_message(Message*)+0xae7) [0x513c57]
  177. 13: (MDS::_dispatch(Message*)+0x33) [0x513d53]
  178. 14: (MDS::ms_dispatch(Message*)+0xab) [0x515b3b]
  179. 15: (DispatchQueue::entry()+0x393) [0x847ca3]
  180. 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7caeed]
  181. 17: (()+0x6b50) [0x7fa3a3376b50]
  182. 18: (clone()+0x6d) [0x7fa3a1d24a7d]
  183. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  184.  
  185. --- logging levels ---
  186. 0/ 5 none
  187. 0/ 1 lockdep
  188. 0/ 1 context
  189. 1/ 1 crush
  190. 1/ 5 mds
  191. 1/ 5 mds_balancer
  192. 1/ 5 mds_locker
  193. 1/ 5 mds_log
  194. 1/ 5 mds_log_expire
  195. 1/ 5 mds_migrator
  196. 0/ 1 buffer
  197. 0/ 1 timer
  198. 0/ 1 filer
  199. 0/ 1 striper
  200. 0/ 1 objecter
  201. 0/ 5 rados
  202. 0/ 5 rbd
  203. 0/ 5 journaler
  204. 0/ 5 objectcacher
  205. 0/ 5 client
  206. 0/ 5 osd
  207. 0/ 5 optracker
  208. 0/ 5 objclass
  209. 1/ 3 filestore
  210. 1/ 3 journal
  211. 0/ 5 ms
  212. 1/ 5 mon
  213. 0/10 monc
  214. 0/ 5 paxos
  215. 0/ 5 tp
  216. 1/ 5 auth
  217. 1/ 5 crypto
  218. 1/ 1 finisher
  219. 1/ 5 heartbeatmap
  220. 1/ 5 perfcounter
  221. 1/ 5 rgw
  222. 1/ 5 hadoop
  223. 1/ 5 javaclient
  224. 1/ 5 asok
  225. 1/ 1 throttle
  226. -2/-2 (syslog threshold)
  227. -1/-1 (stderr threshold)
  228. max_recent 10000
  229. max_new 1000
  230. log_file /var/log/ceph/ceph-mds.stg02.log
  231. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!