API tools faq
paste
alohamora

monitor down

alohamora
Mar 25th, 2015
324
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.89 KB | None | 0 0
raw download clone embed print report
  1. -41> 2015-03-25 14:16:40.509119 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 caps
  2. -40> 2015-03-25 14:16:40.509122 7f6c0b2fe700 10 mon.storage0105@1(leader).auth v905 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from client.8543 10.100.1.31:0/12031720
  3. -39> 2015-03-25 14:16:40.509128 7f6c0b2fe700 10 mon.storage0105@1(leader).auth v905 prep_auth() blob_size=144
  4. -38> 2015-03-25 14:16:40.509365 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 have connection
  5. -37> 2015-03-25 14:16:40.509374 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 ms_dispatch existing session MonSession: mon.2 10.100.1.110:6789/0 is openallow * for mon.2 10.100.1.110:6789/0
  6. -36> 2015-03-25 14:16:40.509383 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 caps allow *
  7. -35> 2015-03-25 14:16:40.509386 7f6c0b2fe700 20 is_capable service=mon command= exec on cap allow *
  8. -34> 2015-03-25 14:16:40.509388 7f6c0b2fe700 20 allow so far , doing grant allow *
  9. -33> 2015-03-25 14:16:40.509389 7f6c0b2fe700 20 allow all
  10. -32> 2015-03-25 14:16:40.510202 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 have connection
  11. -31> 2015-03-25 14:16:40.510212 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 ms_dispatch existing session MonSession: client.8543 10.100.1.31:0/12031720 is openallow r for client.8543 10.100.1.31:0/12031720
  12. -30> 2015-03-25 14:16:40.510221 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 caps allow r
  13. -29> 2015-03-25 14:16:40.510223 7f6c0b2fe700 10 mon.storage0105@1(leader) e3 handle_subscribe mon_subscribe({monmap=4+,osdmap=860}) v2
  14. -28> 2015-03-25 14:16:40.510231 7f6c0b2fe700 10 mon.storage0105@1(leader) e3 check_sub monmap next 4 have 3
  15. -27> 2015-03-25 14:16:40.510235 7f6c0b2fe700 20 is_capable service=osd command= read on cap allow r
  16. -26> 2015-03-25 14:16:40.510237 7f6c0b2fe700 20 allow so far , doing grant allow r
  17. -25> 2015-03-25 14:16:40.510238 7f6c0b2fe700 20 match
  18. -24> 2015-03-25 14:16:40.510240 7f6c0b2fe700 10 mon.storage0105@1(leader).osd e874 check_sub 0x41d89c0 next 860 (onetime)
  19. -23> 2015-03-25 14:16:40.510243 7f6c0b2fe700 5 mon.storage0105@1(leader).osd e874 send_incremental [860..874] to client.8543 10.100.1.31:0/12031720
  20. -22> 2015-03-25 14:16:40.510248 7f6c0b2fe700 10 mon.storage0105@1(leader).osd e874 build_incremental [860..874]
  21. -21> 2015-03-25 14:16:40.510287 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 874 174 bytes
  22. -20> 2015-03-25 14:16:40.510311 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 873 215 bytes
  23. -19> 2015-03-25 14:16:40.510331 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 872 239 bytes
  24. -18> 2015-03-25 14:16:40.510351 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 871 182 bytes
  25. -17> 2015-03-25 14:16:40.510370 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 870 190 bytes
  26. -16> 2015-03-25 14:16:40.510391 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 869 223 bytes
  27. -15> 2015-03-25 14:16:40.510411 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 868 215 bytes
  28. -14> 2015-03-25 14:16:40.510431 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 867 174 bytes
  29. -13> 2015-03-25 14:16:40.510450 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 866 224 bytes
  30. -12> 2015-03-25 14:16:40.510469 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 865 174 bytes
  31. -11> 2015-03-25 14:16:40.510488 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 864 174 bytes
  32. -10> 2015-03-25 14:16:40.510518 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 863 174 bytes
  33. -9> 2015-03-25 14:16:40.510539 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 862 182 bytes
  34. -8> 2015-03-25 14:16:40.510559 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 861 190 bytes
  35. -7> 2015-03-25 14:16:40.510578 7f6c0b2fe700 20 mon.storage0105@1(leader).osd e874 build_incremental inc 860 350 bytes
  36. -6> 2015-03-25 14:16:40.510596 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 have connection
  37. -5> 2015-03-25 14:16:40.510602 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 ms_dispatch existing session MonSession: client.8543 10.100.1.31:0/12031720 is openallow r for client.8543 10.100.1.31:0/12031720
  38. -4> 2015-03-25 14:16:40.510610 7f6c0b2fe700 20 mon.storage0105@1(leader) e3 caps allow r
  39. -3> 2015-03-25 14:16:40.510613 7f6c0b2fe700 10 mon.storage0105@1(leader).osd e874 preprocess_query pool_op(delete unmanaged snap pool 11 auid 0 tid 27 name v0) v4 from client.8543 10.100.1.31:0/12031720
  40. -2> 2015-03-25 14:16:40.510621 7f6c0b2fe700 7 mon.storage0105@1(leader).osd e874 prepare_update pool_op(delete unmanaged snap pool 11 auid 0 tid 27 name v0) v4 from client.8543 10.100.1.31:0/12031720
  41. -1> 2015-03-25 14:16:40.510633 7f6c0b2fe700 10 mon.storage0105@1(leader).osd e874 prepare_pool_op pool_op(delete unmanaged snap pool 11 auid 0 tid 27 name v0) v4
  42. 0> 2015-03-25 14:16:40.511978 7f6c0b2fe700 -1 ./include/interval_set.h: In function 'void interval_set<T>::insert(T, T) [with T = snapid_t]' thread 7f6c0b2fe700 time 2015-03-25 14:16:40.510644
  43. ./include/interval_set.h: 358: FAILED assert(p->first > start+len)
  44.  
  45. ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)
  46. 1: (interval_set<snapid_t>::insert(snapid_t)+0x2a9) [0x70ddb9]
  47. 2: (OSDMonitor::prepare_pool_op(MPoolOp*)+0xd73) [0x5d0913]
  48. 3: (OSDMonitor::prepare_update(PaxosServiceMessage*)+0x20b) [0x5ec02b]
  49. 4: (PaxosService::dispatch(PaxosServiceMessage*)+0xa4a) [0x5b67ba]
  50. 5: (Monitor::dispatch(MonSession*, Message*, bool)+0xf3) [0x583483]
  51. 6: (Monitor::_ms_dispatch(Message*)+0x20e) [0x583c9e]
  52. 7: (Monitor::ms_dispatch(Message*)+0x32) [0x5a0122]
  53. 8: (DispatchQueue::entry()+0x5a2) [0x86a5f2]
  54. 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x82167d]
  55. 10: /lib64/libpthread.so.0() [0x3cc92079d1]
  56. 11: (clone()+0x6d) [0x3cc8ee89dd]
  57. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  58.  
  59. --- logging levels ---
  60. 0/ 5 none
  61. 0/ 0 lockdep
  62. 0/ 0 context
  63. 0/ 0 crush
  64. 1/ 5 mds
  65. 1/ 5 mds_balancer
  66. 1/ 5 mds_locker
  67. 1/ 5 mds_log
  68. 1/ 5 mds_log_expire
  69. 1/ 5 mds_migrator
  70. 0/ 0 buffer
  71. 0/ 0 timer
  72. 0/ 0 filer
  73. 0/ 1 striper
  74. 0/ 0 objecter
  75. 0/ 0 rados
  76. 0/ 0 rbd
  77. 0/ 0 journaler
  78. 0/ 5 objectcacher
  79. 0/ 0 client
  80. 0/ 0 osd
  81. 0/ 0 optracker
  82. 0/ 0 objclass
  83. 0/ 0 filestore
  84. 1/ 3 keyvaluestore
  85. 0/ 0 journal
  86. 0/ 0 ms
  87. 20/ 0 mon
  88. 0/ 0 monc
  89. 0/ 0 paxos
  90. 0/ 0 tp
  91. 0/ 0 auth
  92. 1/ 5 crypto
  93. 0/ 0 finisher
  94. 0/ 0 heartbeatmap
  95. 0/ 0 perfcounter
  96. 0/ 0 rgw
  97. 1/10 civetweb
  98. 1/ 5 javaclient
  99. 0/ 0 asok
  100. 0/ 0 throttle
  101. -2/-2 (syslog threshold)
  102. -1/-1 (stderr threshold)
  103. max_recent 10000
  104. max_new 1000
  105. log_file /var/log/ceph/ceph-mon.storage0105.log
  106. --- end dump of recent events ---
  107. 2015-03-25 14:16:40.649558 7f6c0b2fe700 -1 *** Caught signal (Aborted) **
  108. in thread 7f6c0b2fe700
  109.  
  110. ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)
  111. 1: /usr/bin/ceph-mon() [0x8764f1]
  112. 2: /lib64/libpthread.so.0() [0x3cc920f710]
  113. 3: (gsignal()+0x35) [0x3cc8e32625]
  114. 4: (abort()+0x175) [0x3cc8e33e05]
  115. 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x309a2bea7d]
  116. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  117.  
  118. --- begin dump of recent events ---
  119. 0> 2015-03-25 14:16:40.649558 7f6c0b2fe700 -1 *** Caught signal (Aborted) **
  120. in thread 7f6c0b2fe700
  121.  
  122. ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)
  123. 1: /usr/bin/ceph-mon() [0x8764f1]
  124. 2: /lib64/libpthread.so.0() [0x3cc920f710]
  125. 3: (gsignal()+0x35) [0x3cc8e32625]
  126. 4: (abort()+0x175) [0x3cc8e33e05]
  127. 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x309a2bea7d]
  128. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  129.  
  130. --- logging levels ---
  131. 0/ 5 none
  132. 0/ 0 lockdep
  133. 0/ 0 context
  134. 0/ 0 crush
  135. 1/ 5 mds
  136. 1/ 5 mds_balancer
  137. 1/ 5 mds_locker
  138. 1/ 5 mds_log
  139. 1/ 5 mds_log_expire
  140. 1/ 5 mds_migrator
  141. 0/ 0 buffer
  142. 0/ 0 timer
  143. 0/ 0 filer
  144. 0/ 1 striper
  145. 0/ 0 objecter
  146. 0/ 0 rados
  147. 0/ 0 rbd
  148. 0/ 0 journaler
  149. 0/ 5 objectcacher
  150. 0/ 0 client
  151. 0/ 0 osd
  152. 0/ 0 optracker
  153. 0/ 0 objclass
  154. 0/ 0 filestore
  155. 1/ 3 keyvaluestore
  156. 0/ 0 journal
  157. 0/ 0 ms
  158. 20/ 0 mon
  159. 0/ 0 monc
  160. 0/ 0 paxos
  161. 0/ 0 tp
  162. 0/ 0 auth
  163. 1/ 5 crypto
  164. 0/ 0 finisher
  165. 0/ 0 heartbeatmap
  166. 0/ 0 perfcounter
  167. 0/ 0 rgw
  168. 1/10 civetweb
  169. 1/ 5 javaclient
  170. 0/ 0 asok
  171. 0/ 0 throttle
  172. -2/-2 (syslog threshold)
  173. -1/-1 (stderr threshold)
  174. max_recent 10000
  175. max_new 1000
  176. log_file /var/log/ceph/ceph-mon.storage0105.log
  177. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!