API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jan 15th, 2015
249
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 88.25 KB | None | 0 0
raw download clone embed print report
  1. 2015-01-15 11:34:46.173288 7f214ce6c800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 3019
  2. 2015-01-15 11:34:46.175656 7f214ce6c800 -1 mds.-1.0 log_to_monitors {default=true}
  3. 2015-01-15 11:34:46.255498 7f214ce6c800 -1 mds.-1.0 *** no OSDs are up as of epoch 1, waiting
  4. 2015-01-15 11:34:56.255752 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  5. 2015-01-15 11:35:06.175889 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  6. 2015-01-15 11:35:06.255988 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  7. 2015-01-15 11:35:11.176049 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  8. 2015-01-15 11:35:16.176319 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  9. 2015-01-15 11:35:16.256236 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  10. 2015-01-15 11:35:21.176560 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  11. 2015-01-15 11:35:26.176777 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  12. 2015-01-15 11:35:26.256464 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  13. 2015-01-15 11:35:31.177021 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  14. 2015-01-15 11:35:36.177329 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  15. 2015-01-15 11:35:36.256701 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  16. 2015-01-15 11:35:41.177554 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  17. 2015-01-15 11:35:46.177759 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  18. 2015-01-15 11:35:46.256924 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  19. 2015-01-15 11:35:51.177963 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  20. 2015-01-15 11:35:56.178176 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  21. 2015-01-15 11:35:56.257105 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  22. 2015-01-15 11:36:01.178400 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  23. 2015-01-15 11:36:06.178625 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  24. 2015-01-15 11:36:06.257346 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  25. 2015-01-15 11:36:11.178848 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  26. 2015-01-15 11:36:16.179075 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  27. 2015-01-15 11:36:16.257584 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  28. 2015-01-15 11:36:21.179278 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  29. 2015-01-15 11:36:26.179504 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  30. 2015-01-15 11:36:26.257802 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  31. 2015-01-15 11:36:31.179711 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  32. 2015-01-15 11:36:36.179939 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  33. 2015-01-15 11:36:36.258025 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  34. 2015-01-15 11:36:41.180147 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  35. 2015-01-15 11:36:46.180357 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  36. 2015-01-15 11:36:46.258261 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  37. 2015-01-15 11:36:51.180564 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  38. 2015-01-15 11:36:56.180801 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  39. 2015-01-15 11:36:56.258493 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  40. 2015-01-15 11:37:01.181026 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  41. 2015-01-15 11:37:06.181258 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  42. 2015-01-15 11:37:06.258726 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  43. 2015-01-15 11:37:11.181460 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  44. 2015-01-15 11:37:16.181686 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  45. 2015-01-15 11:37:16.258958 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 1, waiting
  46. 2015-01-15 11:37:21.181892 7f214925e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15
  47. 2015-01-15 11:37:23.078404 7f214725a700 1 heartbeat_map reset_timeout 'MDS' had timed out after 15
  48. 2015-01-15 11:37:26.259204 7f214ce6c800 -1 mds.-1.-1 *** no OSDs are up as of epoch 3, waiting
  49. 2015-01-15 13:14:23.361861 7f2142950700 -1 mds.-1.-1 *** got signal Terminated ***
  50. 2015-01-15 13:14:23.361901 7f2142950700 1 mds.-1.-1 suicide. wanted down:dne, now down:dne
  51. 2015-01-15 13:14:23.385996 7f48f424d800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6717
  52. 2015-01-15 13:14:23.388258 7f48f424d800 -1 mds.-1.0 log_to_monitors {default=true}
  53. 2015-01-15 13:14:23.390817 7f48f424d800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  54. 2015-01-15 13:14:23.390839 7f48f424d800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  55. 2015-01-15 13:14:23.392899 7f48f424d800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f48f424d800 time 2015-01-15 13:14:23.391350
  56. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  57.  
  58. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  59. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  60. 2: (Thread::join(void**)+0x41) [0x9308a1]
  61. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  62. 4: (MDS::suicide()+0x117) [0x59bc97]
  63. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  64. 6: (main()+0x94d) [0x59038d]
  65. 7: (__libc_start_main()+0xf5) [0x7f48f2251ec5]
  66. 8: /usr/bin/ceph-mds() [0x594d87]
  67. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  68.  
  69. --- begin dump of recent events ---
  70. -70> 2015-01-15 13:14:23.384158 7f48f424d800 5 asok(0x4588000) register_command perfcounters_dump hook 0x4568050
  71. -69> 2015-01-15 13:14:23.384186 7f48f424d800 5 asok(0x4588000) register_command 1 hook 0x4568050
  72. -68> 2015-01-15 13:14:23.384189 7f48f424d800 5 asok(0x4588000) register_command perf dump hook 0x4568050
  73. -67> 2015-01-15 13:14:23.384197 7f48f424d800 5 asok(0x4588000) register_command perfcounters_schema hook 0x4568050
  74. -66> 2015-01-15 13:14:23.384201 7f48f424d800 5 asok(0x4588000) register_command 2 hook 0x4568050
  75. -65> 2015-01-15 13:14:23.384203 7f48f424d800 5 asok(0x4588000) register_command perf schema hook 0x4568050
  76. -64> 2015-01-15 13:14:23.384206 7f48f424d800 5 asok(0x4588000) register_command perf reset hook 0x4568050
  77. -63> 2015-01-15 13:14:23.384208 7f48f424d800 5 asok(0x4588000) register_command config show hook 0x4568050
  78. -62> 2015-01-15 13:14:23.384321 7f48f424d800 5 asok(0x4588000) register_command config set hook 0x4568050
  79. -61> 2015-01-15 13:14:23.384346 7f48f424d800 5 asok(0x4588000) register_command config get hook 0x4568050
  80. -60> 2015-01-15 13:14:23.384352 7f48f424d800 5 asok(0x4588000) register_command config diff hook 0x4568050
  81. -59> 2015-01-15 13:14:23.384361 7f48f424d800 5 asok(0x4588000) register_command log flush hook 0x4568050
  82. -58> 2015-01-15 13:14:23.384370 7f48f424d800 5 asok(0x4588000) register_command log dump hook 0x4568050
  83. -57> 2015-01-15 13:14:23.384377 7f48f424d800 5 asok(0x4588000) register_command log reopen hook 0x4568050
  84. -56> 2015-01-15 13:14:23.385996 7f48f424d800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6717
  85. -55> 2015-01-15 13:14:23.386348 7f48f424d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6717 need_addr=1
  86. -54> 2015-01-15 13:14:23.387589 7f48f424d800 5 asok(0x4588000) init /var/run/ceph/ceph-mds.node02d.asok
  87. -53> 2015-01-15 13:14:23.387600 7f48f424d800 5 asok(0x4588000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  88. -52> 2015-01-15 13:14:23.387621 7f48f424d800 5 asok(0x4588000) register_command 0 hook 0x45640b8
  89. -51> 2015-01-15 13:14:23.387629 7f48f424d800 5 asok(0x4588000) register_command version hook 0x45640b8
  90. -50> 2015-01-15 13:14:23.387632 7f48f424d800 5 asok(0x4588000) register_command git_version hook 0x45640b8
  91. -49> 2015-01-15 13:14:23.387635 7f48f424d800 5 asok(0x4588000) register_command help hook 0x45680b0
  92. -48> 2015-01-15 13:14:23.387645 7f48f424d800 5 asok(0x4588000) register_command get_command_descriptions hook 0x4568150
  93. -47> 2015-01-15 13:14:23.387683 7f48f424d800 10 monclient(hunting): build_initial_monmap
  94. -46> 2015-01-15 13:14:23.387724 7f48f424d800 1 -- 0.0.0.0:6800/6717 messenger.start
  95. -45> 2015-01-15 13:14:23.387759 7f48f424d800 5 adding auth protocol: cephx
  96. -44> 2015-01-15 13:14:23.387764 7f48f424d800 5 adding auth protocol: cephx
  97. -43> 2015-01-15 13:14:23.387790 7f48efe3e700 5 asok(0x4588000) entry start
  98. -42> 2015-01-15 13:14:23.387850 7f48f424d800 5 asok(0x4588000) register_command objecter_requests hook 0x4568140
  99. -41> 2015-01-15 13:14:23.387920 7f48f424d800 1 accepter.accepter.start
  100. -40> 2015-01-15 13:14:23.388023 7f48f424d800 10 monclient(hunting): init
  101. -39> 2015-01-15 13:14:23.388040 7f48f424d800 5 adding auth protocol: cephx
  102. -38> 2015-01-15 13:14:23.388043 7f48f424d800 10 monclient(hunting): auth_supported 2 method cephx
  103. -37> 2015-01-15 13:14:23.388161 7f48f424d800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  104. -36> 2015-01-15 13:14:23.388252 7f48f424d800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  105. -35> 2015-01-15 13:14:23.388258 7f48f424d800 -1 mds.-1.0 log_to_monitors {default=true}
  106. -34> 2015-01-15 13:14:23.388274 7f48f424d800 10 monclient(hunting): _reopen_session rank -1 name
  107. -33> 2015-01-15 13:14:23.388323 7f48f424d800 10 monclient(hunting): picked mon.noname-e con 0x459ef20 addr 10.0.0.6:6789/0
  108. -32> 2015-01-15 13:14:23.388341 7f48f424d800 10 monclient(hunting): _send_mon_message to mon.noname-e at 10.0.0.6:6789/0
  109. -31> 2015-01-15 13:14:23.388347 7f48f424d800 1 -- 0.0.0.0:6800/6717 --> 10.0.0.6:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x4584000 con 0x459ef20
  110. -30> 2015-01-15 13:14:23.388360 7f48f424d800 10 monclient(hunting): renew_subs
  111. -29> 2015-01-15 13:14:23.389503 7f48f424b700 1 -- 10.0.0.6:6800/6717 learned my addr 10.0.0.6:6800/6717
  112. -28> 2015-01-15 13:14:23.390077 7f48ee63b700 1 -- 10.0.0.6:6800/6717 <== mon.4 10.0.0.6:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x4597400 con 0x459ef20
  113. -27> 2015-01-15 13:14:23.390111 7f48ee63b700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  114. -26> 2015-01-15 13:14:23.390133 7f48ee63b700 10 monclient(hunting): got monmap 2, mon.noname-e is now rank -1
  115. -25> 2015-01-15 13:14:23.390138 7f48ee63b700 10 monclient(hunting): dump:
  116. epoch 2
  117. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  118. last_changed 2015-01-15 11:34:08.529401
  119. created 0.000000
  120. 0: 10.0.0.2:6789/0 mon.node01
  121. 1: 10.0.0.3:6789/0 mon.node02a
  122. 2: 10.0.0.4:6789/0 mon.node02b
  123. 3: 10.0.0.5:6789/0 mon.node02c
  124. 4: 10.0.0.6:6789/0 mon.node02d
  125.  
  126. -24> 2015-01-15 13:14:23.390170 7f48ee63b700 1 -- 10.0.0.6:6800/6717 <== mon.4 10.0.0.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3262903505 0 0) 0x4584480 con 0x459ef20
  127. -23> 2015-01-15 13:14:23.390194 7f48ee63b700 10 monclient(hunting): my global_id is 4339
  128. -22> 2015-01-15 13:14:23.390237 7f48ee63b700 10 monclient(hunting): _send_mon_message to mon.node02d at 10.0.0.6:6789/0
  129. -21> 2015-01-15 13:14:23.390245 7f48ee63b700 1 -- 10.0.0.6:6800/6717 --> 10.0.0.6:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x45846c0 con 0x459ef20
  130. -20> 2015-01-15 13:14:23.390775 7f48ee63b700 1 -- 10.0.0.6:6800/6717 <== mon.4 10.0.0.6:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (2532590124 0 0) 0x45846c0 con 0x459ef20
  131. -19> 2015-01-15 13:14:23.390803 7f48ee63b700 1 monclient(hunting): found mon.node02d
  132. -18> 2015-01-15 13:14:23.390817 7f48f424d800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  133. -17> 2015-01-15 13:14:23.390839 7f48f424d800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  134. -16> 2015-01-15 13:14:23.390844 7f48f424d800 5 mds.-1.log shutdown
  135. -15> 2015-01-15 13:14:23.390956 7f48f424d800 5 asok(0x4588000) unregister_command status ENOENT
  136. -14> 2015-01-15 13:14:23.390962 7f48f424d800 5 asok(0x4588000) unregister_command dump_ops_in_flight ENOENT
  137. -13> 2015-01-15 13:14:23.390964 7f48f424d800 5 asok(0x4588000) unregister_command ops ENOENT
  138. -12> 2015-01-15 13:14:23.390968 7f48f424d800 5 asok(0x4588000) unregister_command dump_historic_ops ENOENT
  139. -11> 2015-01-15 13:14:23.390970 7f48f424d800 5 asok(0x4588000) unregister_command scrub_path ENOENT
  140. -10> 2015-01-15 13:14:23.390973 7f48f424d800 5 asok(0x4588000) unregister_command flush_path ENOENT
  141. -9> 2015-01-15 13:14:23.390974 7f48f424d800 5 asok(0x4588000) unregister_command session evict ENOENT
  142. -8> 2015-01-15 13:14:23.390975 7f48f424d800 5 asok(0x4588000) unregister_command session ls ENOENT
  143. -7> 2015-01-15 13:14:23.390977 7f48f424d800 5 asok(0x4588000) unregister_command flush journal ENOENT
  144. -6> 2015-01-15 13:14:23.390978 7f48f424d800 5 asok(0x4588000) unregister_command force_readonly ENOENT
  145. -5> 2015-01-15 13:14:23.390984 7f48f424d800 5 asok(0x4588000) unregister_command objecter_requests
  146. -4> 2015-01-15 13:14:23.391116 7f48f424d800 10 monclient: shutdown
  147. -3> 2015-01-15 13:14:23.391304 7f48f424d800 1 -- 10.0.0.6:6800/6717 mark_down 0x459ef20 -- 0x4604000
  148. -2> 2015-01-15 13:14:23.391348 7f48eb635700 2 -- 10.0.0.6:6800/6717 >> 10.0.0.6:6789/0 pipe(0x4604000 sd=8 :45303 s=4 pgs=118 cs=1 l=1 c=0x459ef20).reader couldn't read tag, (0) Success
  149. -1> 2015-01-15 13:14:23.391394 7f48eb635700 2 -- 10.0.0.6:6800/6717 >> 10.0.0.6:6789/0 pipe(0x4604000 sd=8 :45303 s=4 pgs=118 cs=1 l=1 c=0x459ef20).fault (0) Success
  150. 0> 2015-01-15 13:14:23.392899 7f48f424d800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f48f424d800 time 2015-01-15 13:14:23.391350
  151. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  152.  
  153. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  154. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  155. 2: (Thread::join(void**)+0x41) [0x9308a1]
  156. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  157. 4: (MDS::suicide()+0x117) [0x59bc97]
  158. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  159. 6: (main()+0x94d) [0x59038d]
  160. 7: (__libc_start_main()+0xf5) [0x7f48f2251ec5]
  161. 8: /usr/bin/ceph-mds() [0x594d87]
  162. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  163.  
  164. --- logging levels ---
  165. 0/ 5 none
  166. 0/ 1 lockdep
  167. 0/ 1 context
  168. 1/ 1 crush
  169. 1/ 5 mds
  170. 1/ 5 mds_balancer
  171. 1/ 5 mds_locker
  172. 1/ 5 mds_log
  173. 1/ 5 mds_log_expire
  174. 1/ 5 mds_migrator
  175. 0/ 1 buffer
  176. 0/ 1 timer
  177. 0/ 1 filer
  178. 0/ 1 striper
  179. 0/ 1 objecter
  180. 0/ 5 rados
  181. 0/ 5 rbd
  182. 0/ 5 rbd_replay
  183. 0/ 5 journaler
  184. 0/ 5 objectcacher
  185. 0/ 5 client
  186. 0/ 5 osd
  187. 0/ 5 optracker
  188. 0/ 5 objclass
  189. 1/ 3 filestore
  190. 1/ 3 keyvaluestore
  191. 1/ 3 journal
  192. 0/ 5 ms
  193. 1/ 5 mon
  194. 0/10 monc
  195. 1/ 5 paxos
  196. 0/ 5 tp
  197. 1/ 5 auth
  198. 1/ 5 crypto
  199. 1/ 1 finisher
  200. 1/ 5 heartbeatmap
  201. 1/ 5 perfcounter
  202. 1/ 5 rgw
  203. 1/10 civetweb
  204. 1/ 5 javaclient
  205. 1/ 5 asok
  206. 1/ 1 throttle
  207. 0/ 0 refs
  208. -2/-2 (syslog threshold)
  209. -1/-1 (stderr threshold)
  210. max_recent 10000
  211. max_new 1000
  212. log_file /var/log/ceph/ceph-mds.node02d.log
  213. --- end dump of recent events ---
  214. 2015-01-15 13:14:23.395490 7f48f424d800 -1 *** Caught signal (Aborted) **
  215. in thread 7f48f424d800
  216.  
  217. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  218. 1: /usr/bin/ceph-mds() [0x85721a]
  219. 2: (()+0x10340) [0x7f48f3bc4340]
  220. 3: (gsignal()+0x39) [0x7f48f2266cc9]
  221. 4: (abort()+0x148) [0x7f48f226a0d8]
  222. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f48f2b726b5]
  223. 6: (()+0x5e836) [0x7f48f2b70836]
  224. 7: (()+0x5e863) [0x7f48f2b70863]
  225. 8: (()+0x5eaa2) [0x7f48f2b70aa2]
  226. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  227. 10: (Thread::join(void**)+0x41) [0x9308a1]
  228. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  229. 12: (MDS::suicide()+0x117) [0x59bc97]
  230. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  231. 14: (main()+0x94d) [0x59038d]
  232. 15: (__libc_start_main()+0xf5) [0x7f48f2251ec5]
  233. 16: /usr/bin/ceph-mds() [0x594d87]
  234. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  235.  
  236. --- begin dump of recent events ---
  237. 0> 2015-01-15 13:14:23.395490 7f48f424d800 -1 *** Caught signal (Aborted) **
  238. in thread 7f48f424d800
  239.  
  240. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  241. 1: /usr/bin/ceph-mds() [0x85721a]
  242. 2: (()+0x10340) [0x7f48f3bc4340]
  243. 3: (gsignal()+0x39) [0x7f48f2266cc9]
  244. 4: (abort()+0x148) [0x7f48f226a0d8]
  245. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f48f2b726b5]
  246. 6: (()+0x5e836) [0x7f48f2b70836]
  247. 7: (()+0x5e863) [0x7f48f2b70863]
  248. 8: (()+0x5eaa2) [0x7f48f2b70aa2]
  249. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  250. 10: (Thread::join(void**)+0x41) [0x9308a1]
  251. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  252. 12: (MDS::suicide()+0x117) [0x59bc97]
  253. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  254. 14: (main()+0x94d) [0x59038d]
  255. 15: (__libc_start_main()+0xf5) [0x7f48f2251ec5]
  256. 16: /usr/bin/ceph-mds() [0x594d87]
  257. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  258.  
  259. --- logging levels ---
  260. 0/ 5 none
  261. 0/ 1 lockdep
  262. 0/ 1 context
  263. 1/ 1 crush
  264. 1/ 5 mds
  265. 1/ 5 mds_balancer
  266. 1/ 5 mds_locker
  267. 1/ 5 mds_log
  268. 1/ 5 mds_log_expire
  269. 1/ 5 mds_migrator
  270. 0/ 1 buffer
  271. 0/ 1 timer
  272. 0/ 1 filer
  273. 0/ 1 striper
  274. 0/ 1 objecter
  275. 0/ 5 rados
  276. 0/ 5 rbd
  277. 0/ 5 rbd_replay
  278. 0/ 5 journaler
  279. 0/ 5 objectcacher
  280. 0/ 5 client
  281. 0/ 5 osd
  282. 0/ 5 optracker
  283. 0/ 5 objclass
  284. 1/ 3 filestore
  285. 1/ 3 keyvaluestore
  286. 1/ 3 journal
  287. 0/ 5 ms
  288. 1/ 5 mon
  289. 0/10 monc
  290. 1/ 5 paxos
  291. 0/ 5 tp
  292. 1/ 5 auth
  293. 1/ 5 crypto
  294. 1/ 1 finisher
  295. 1/ 5 heartbeatmap
  296. 1/ 5 perfcounter
  297. 1/ 5 rgw
  298. 1/10 civetweb
  299. 1/ 5 javaclient
  300. 1/ 5 asok
  301. 1/ 1 throttle
  302. 0/ 0 refs
  303. -2/-2 (syslog threshold)
  304. -1/-1 (stderr threshold)
  305. max_recent 10000
  306. max_new 1000
  307. log_file /var/log/ceph/ceph-mds.node02d.log
  308. --- end dump of recent events ---
  309. 2015-01-15 13:14:23.407907 7ff61cce0800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6738
  310. 2015-01-15 13:14:23.410319 7ff61cce0800 -1 mds.-1.0 log_to_monitors {default=true}
  311. 2015-01-15 13:14:23.413001 7ff61cce0800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  312. 2015-01-15 13:14:23.413030 7ff61cce0800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  313. 2015-01-15 13:14:23.415130 7ff61cce0800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7ff61cce0800 time 2015-01-15 13:14:23.413558
  314. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  315.  
  316. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  317. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  318. 2: (Thread::join(void**)+0x41) [0x9308a1]
  319. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  320. 4: (MDS::suicide()+0x117) [0x59bc97]
  321. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  322. 6: (main()+0x94d) [0x59038d]
  323. 7: (__libc_start_main()+0xf5) [0x7ff61ace4ec5]
  324. 8: /usr/bin/ceph-mds() [0x594d87]
  325. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  326.  
  327. --- begin dump of recent events ---
  328. -70> 2015-01-15 13:14:23.406164 7ff61cce0800 5 asok(0x3e00000) register_command perfcounters_dump hook 0x3de0050
  329. -69> 2015-01-15 13:14:23.406201 7ff61cce0800 5 asok(0x3e00000) register_command 1 hook 0x3de0050
  330. -68> 2015-01-15 13:14:23.406206 7ff61cce0800 5 asok(0x3e00000) register_command perf dump hook 0x3de0050
  331. -67> 2015-01-15 13:14:23.406226 7ff61cce0800 5 asok(0x3e00000) register_command perfcounters_schema hook 0x3de0050
  332. -66> 2015-01-15 13:14:23.406232 7ff61cce0800 5 asok(0x3e00000) register_command 2 hook 0x3de0050
  333. -65> 2015-01-15 13:14:23.406234 7ff61cce0800 5 asok(0x3e00000) register_command perf schema hook 0x3de0050
  334. -64> 2015-01-15 13:14:23.406237 7ff61cce0800 5 asok(0x3e00000) register_command perf reset hook 0x3de0050
  335. -63> 2015-01-15 13:14:23.406239 7ff61cce0800 5 asok(0x3e00000) register_command config show hook 0x3de0050
  336. -62> 2015-01-15 13:14:23.406257 7ff61cce0800 5 asok(0x3e00000) register_command config set hook 0x3de0050
  337. -61> 2015-01-15 13:14:23.406266 7ff61cce0800 5 asok(0x3e00000) register_command config get hook 0x3de0050
  338. -60> 2015-01-15 13:14:23.406268 7ff61cce0800 5 asok(0x3e00000) register_command config diff hook 0x3de0050
  339. -59> 2015-01-15 13:14:23.406270 7ff61cce0800 5 asok(0x3e00000) register_command log flush hook 0x3de0050
  340. -58> 2015-01-15 13:14:23.406272 7ff61cce0800 5 asok(0x3e00000) register_command log dump hook 0x3de0050
  341. -57> 2015-01-15 13:14:23.406275 7ff61cce0800 5 asok(0x3e00000) register_command log reopen hook 0x3de0050
  342. -56> 2015-01-15 13:14:23.407907 7ff61cce0800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6738
  343. -55> 2015-01-15 13:14:23.408335 7ff61cce0800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6738 need_addr=1
  344. -54> 2015-01-15 13:14:23.409643 7ff61cce0800 5 asok(0x3e00000) init /var/run/ceph/ceph-mds.node02d.asok
  345. -53> 2015-01-15 13:14:23.409654 7ff61cce0800 5 asok(0x3e00000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  346. -52> 2015-01-15 13:14:23.409747 7ff61cce0800 5 asok(0x3e00000) register_command 0 hook 0x3ddc0b8
  347. -51> 2015-01-15 13:14:23.409810 7ff61cce0800 5 asok(0x3e00000) register_command version hook 0x3ddc0b8
  348. -50> 2015-01-15 13:14:23.409813 7ff61cce0800 5 asok(0x3e00000) register_command git_version hook 0x3ddc0b8
  349. -49> 2015-01-15 13:14:23.409816 7ff61cce0800 5 asok(0x3e00000) register_command help hook 0x3de00b0
  350. -48> 2015-01-15 13:14:23.409818 7ff61cce0800 5 asok(0x3e00000) register_command get_command_descriptions hook 0x3de0150
  351. -47> 2015-01-15 13:14:23.409851 7ff61cce0800 10 monclient(hunting): build_initial_monmap
  352. -46> 2015-01-15 13:14:23.409861 7ff6188d1700 5 asok(0x3e00000) entry start
  353. -45> 2015-01-15 13:14:23.409892 7ff61cce0800 1 -- 0.0.0.0:6800/6738 messenger.start
  354. -44> 2015-01-15 13:14:23.409940 7ff61cce0800 5 adding auth protocol: cephx
  355. -43> 2015-01-15 13:14:23.409946 7ff61cce0800 5 adding auth protocol: cephx
  356. -42> 2015-01-15 13:14:23.410040 7ff61cce0800 5 asok(0x3e00000) register_command objecter_requests hook 0x3de0140
  357. -41> 2015-01-15 13:14:23.410102 7ff61cce0800 1 accepter.accepter.start
  358. -40> 2015-01-15 13:14:23.410125 7ff61cce0800 10 monclient(hunting): init
  359. -39> 2015-01-15 13:14:23.410133 7ff61cce0800 5 adding auth protocol: cephx
  360. -38> 2015-01-15 13:14:23.410134 7ff61cce0800 10 monclient(hunting): auth_supported 2 method cephx
  361. -37> 2015-01-15 13:14:23.410231 7ff61cce0800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  362. -36> 2015-01-15 13:14:23.410316 7ff61cce0800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  363. -35> 2015-01-15 13:14:23.410319 7ff61cce0800 -1 mds.-1.0 log_to_monitors {default=true}
  364. -34> 2015-01-15 13:14:23.410338 7ff61cce0800 10 monclient(hunting): _reopen_session rank -1 name
  365. -33> 2015-01-15 13:14:23.410380 7ff61cce0800 10 monclient(hunting): picked mon.noname-c con 0x3e16f20 addr 10.0.0.4:6789/0
  366. -32> 2015-01-15 13:14:23.410398 7ff61cce0800 10 monclient(hunting): _send_mon_message to mon.noname-c at 10.0.0.4:6789/0
  367. -31> 2015-01-15 13:14:23.410402 7ff61cce0800 1 -- 0.0.0.0:6800/6738 --> 10.0.0.4:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x3dfc000 con 0x3e16f20
  368. -30> 2015-01-15 13:14:23.410414 7ff61cce0800 10 monclient(hunting): renew_subs
  369. -29> 2015-01-15 13:14:23.411088 7ff61ccde700 1 -- 10.0.0.6:6800/6738 learned my addr 10.0.0.6:6800/6738
  370. -28> 2015-01-15 13:14:23.412068 7ff6170ce700 1 -- 10.0.0.6:6800/6738 <== mon.2 10.0.0.4:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x3e0f400 con 0x3e16f20
  371. -27> 2015-01-15 13:14:23.412118 7ff6170ce700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  372. -26> 2015-01-15 13:14:23.412134 7ff6170ce700 10 monclient(hunting): got monmap 2, mon.noname-c is now rank -1
  373. -25> 2015-01-15 13:14:23.412136 7ff6170ce700 10 monclient(hunting): dump:
  374. epoch 2
  375. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  376. last_changed 2015-01-15 11:34:08.529401
  377. created 0.000000
  378. 0: 10.0.0.2:6789/0 mon.node01
  379. 1: 10.0.0.3:6789/0 mon.node02a
  380. 2: 10.0.0.4:6789/0 mon.node02b
  381. 3: 10.0.0.5:6789/0 mon.node02c
  382. 4: 10.0.0.6:6789/0 mon.node02d
  383.  
  384. -24> 2015-01-15 13:14:23.412173 7ff6170ce700 1 -- 10.0.0.6:6800/6738 <== mon.2 10.0.0.4:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (468760723 0 0) 0x3dfc480 con 0x3e16f20
  385. -23> 2015-01-15 13:14:23.412197 7ff6170ce700 10 monclient(hunting): my global_id is 4237
  386. -22> 2015-01-15 13:14:23.412253 7ff6170ce700 10 monclient(hunting): _send_mon_message to mon.node02b at 10.0.0.4:6789/0
  387. -21> 2015-01-15 13:14:23.412259 7ff6170ce700 1 -- 10.0.0.6:6800/6738 --> 10.0.0.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3dfc6c0 con 0x3e16f20
  388. -20> 2015-01-15 13:14:23.412866 7ff6170ce700 1 -- 10.0.0.6:6800/6738 <== mon.2 10.0.0.4:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (1511225296 0 0) 0x3dfc6c0 con 0x3e16f20
  389. -19> 2015-01-15 13:14:23.412909 7ff6170ce700 1 monclient(hunting): found mon.node02b
  390. -18> 2015-01-15 13:14:23.413001 7ff61cce0800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  391. -17> 2015-01-15 13:14:23.413030 7ff61cce0800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  392. -16> 2015-01-15 13:14:23.413036 7ff61cce0800 5 mds.-1.log shutdown
  393. -15> 2015-01-15 13:14:23.413206 7ff61cce0800 5 asok(0x3e00000) unregister_command status ENOENT
  394. -14> 2015-01-15 13:14:23.413216 7ff61cce0800 5 asok(0x3e00000) unregister_command dump_ops_in_flight ENOENT
  395. -13> 2015-01-15 13:14:23.413218 7ff61cce0800 5 asok(0x3e00000) unregister_command ops ENOENT
  396. -12> 2015-01-15 13:14:23.413220 7ff61cce0800 5 asok(0x3e00000) unregister_command dump_historic_ops ENOENT
  397. -11> 2015-01-15 13:14:23.413222 7ff61cce0800 5 asok(0x3e00000) unregister_command scrub_path ENOENT
  398. -10> 2015-01-15 13:14:23.413223 7ff61cce0800 5 asok(0x3e00000) unregister_command flush_path ENOENT
  399. -9> 2015-01-15 13:14:23.413225 7ff61cce0800 5 asok(0x3e00000) unregister_command session evict ENOENT
  400. -8> 2015-01-15 13:14:23.413226 7ff61cce0800 5 asok(0x3e00000) unregister_command session ls ENOENT
  401. -7> 2015-01-15 13:14:23.413227 7ff61cce0800 5 asok(0x3e00000) unregister_command flush journal ENOENT
  402. -6> 2015-01-15 13:14:23.413229 7ff61cce0800 5 asok(0x3e00000) unregister_command force_readonly ENOENT
  403. -5> 2015-01-15 13:14:23.413237 7ff61cce0800 5 asok(0x3e00000) unregister_command objecter_requests
  404. -4> 2015-01-15 13:14:23.413322 7ff61cce0800 10 monclient: shutdown
  405. -3> 2015-01-15 13:14:23.413505 7ff61cce0800 1 -- 10.0.0.6:6800/6738 mark_down 0x3e16f20 -- 0x3e8c000
  406. -2> 2015-01-15 13:14:23.413552 7ff6140c8700 2 -- 10.0.0.6:6800/6738 >> 10.0.0.4:6789/0 pipe(0x3e8c000 sd=8 :50904 s=4 pgs=922 cs=1 l=1 c=0x3e16f20).reader couldn't read tag, (0) Success
  407. -1> 2015-01-15 13:14:23.413604 7ff6140c8700 2 -- 10.0.0.6:6800/6738 >> 10.0.0.4:6789/0 pipe(0x3e8c000 sd=8 :50904 s=4 pgs=922 cs=1 l=1 c=0x3e16f20).fault (0) Success
  408. 0> 2015-01-15 13:14:23.415130 7ff61cce0800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7ff61cce0800 time 2015-01-15 13:14:23.413558
  409. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  410.  
  411. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  412. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  413. 2: (Thread::join(void**)+0x41) [0x9308a1]
  414. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  415. 4: (MDS::suicide()+0x117) [0x59bc97]
  416. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  417. 6: (main()+0x94d) [0x59038d]
  418. 7: (__libc_start_main()+0xf5) [0x7ff61ace4ec5]
  419. 8: /usr/bin/ceph-mds() [0x594d87]
  420. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  421.  
  422. --- logging levels ---
  423. 0/ 5 none
  424. 0/ 1 lockdep
  425. 0/ 1 context
  426. 1/ 1 crush
  427. 1/ 5 mds
  428. 1/ 5 mds_balancer
  429. 1/ 5 mds_locker
  430. 1/ 5 mds_log
  431. 1/ 5 mds_log_expire
  432. 1/ 5 mds_migrator
  433. 0/ 1 buffer
  434. 0/ 1 timer
  435. 0/ 1 filer
  436. 0/ 1 striper
  437. 0/ 1 objecter
  438. 0/ 5 rados
  439. 0/ 5 rbd
  440. 0/ 5 rbd_replay
  441. 0/ 5 journaler
  442. 0/ 5 objectcacher
  443. 0/ 5 client
  444. 0/ 5 osd
  445. 0/ 5 optracker
  446. 0/ 5 objclass
  447. 1/ 3 filestore
  448. 1/ 3 keyvaluestore
  449. 1/ 3 journal
  450. 0/ 5 ms
  451. 1/ 5 mon
  452. 0/10 monc
  453. 1/ 5 paxos
  454. 0/ 5 tp
  455. 1/ 5 auth
  456. 1/ 5 crypto
  457. 1/ 1 finisher
  458. 1/ 5 heartbeatmap
  459. 1/ 5 perfcounter
  460. 1/ 5 rgw
  461. 1/10 civetweb
  462. 1/ 5 javaclient
  463. 1/ 5 asok
  464. 1/ 1 throttle
  465. 0/ 0 refs
  466. -2/-2 (syslog threshold)
  467. -1/-1 (stderr threshold)
  468. max_recent 10000
  469. max_new 1000
  470. log_file /var/log/ceph/ceph-mds.node02d.log
  471. --- end dump of recent events ---
  472. 2015-01-15 13:14:23.417688 7ff61cce0800 -1 *** Caught signal (Aborted) **
  473. in thread 7ff61cce0800
  474.  
  475. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  476. 1: /usr/bin/ceph-mds() [0x85721a]
  477. 2: (()+0x10340) [0x7ff61c657340]
  478. 3: (gsignal()+0x39) [0x7ff61acf9cc9]
  479. 4: (abort()+0x148) [0x7ff61acfd0d8]
  480. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff61b6056b5]
  481. 6: (()+0x5e836) [0x7ff61b603836]
  482. 7: (()+0x5e863) [0x7ff61b603863]
  483. 8: (()+0x5eaa2) [0x7ff61b603aa2]
  484. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  485. 10: (Thread::join(void**)+0x41) [0x9308a1]
  486. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  487. 12: (MDS::suicide()+0x117) [0x59bc97]
  488. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  489. 14: (main()+0x94d) [0x59038d]
  490. 15: (__libc_start_main()+0xf5) [0x7ff61ace4ec5]
  491. 16: /usr/bin/ceph-mds() [0x594d87]
  492. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  493.  
  494. --- begin dump of recent events ---
  495. 0> 2015-01-15 13:14:23.417688 7ff61cce0800 -1 *** Caught signal (Aborted) **
  496. in thread 7ff61cce0800
  497.  
  498. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  499. 1: /usr/bin/ceph-mds() [0x85721a]
  500. 2: (()+0x10340) [0x7ff61c657340]
  501. 3: (gsignal()+0x39) [0x7ff61acf9cc9]
  502. 4: (abort()+0x148) [0x7ff61acfd0d8]
  503. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff61b6056b5]
  504. 6: (()+0x5e836) [0x7ff61b603836]
  505. 7: (()+0x5e863) [0x7ff61b603863]
  506. 8: (()+0x5eaa2) [0x7ff61b603aa2]
  507. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  508. 10: (Thread::join(void**)+0x41) [0x9308a1]
  509. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  510. 12: (MDS::suicide()+0x117) [0x59bc97]
  511. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  512. 14: (main()+0x94d) [0x59038d]
  513. 15: (__libc_start_main()+0xf5) [0x7ff61ace4ec5]
  514. 16: /usr/bin/ceph-mds() [0x594d87]
  515. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  516.  
  517. --- logging levels ---
  518. 0/ 5 none
  519. 0/ 1 lockdep
  520. 0/ 1 context
  521. 1/ 1 crush
  522. 1/ 5 mds
  523. 1/ 5 mds_balancer
  524. 1/ 5 mds_locker
  525. 1/ 5 mds_log
  526. 1/ 5 mds_log_expire
  527. 1/ 5 mds_migrator
  528. 0/ 1 buffer
  529. 0/ 1 timer
  530. 0/ 1 filer
  531. 0/ 1 striper
  532. 0/ 1 objecter
  533. 0/ 5 rados
  534. 0/ 5 rbd
  535. 0/ 5 rbd_replay
  536. 0/ 5 journaler
  537. 0/ 5 objectcacher
  538. 0/ 5 client
  539. 0/ 5 osd
  540. 0/ 5 optracker
  541. 0/ 5 objclass
  542. 1/ 3 filestore
  543. 1/ 3 keyvaluestore
  544. 1/ 3 journal
  545. 0/ 5 ms
  546. 1/ 5 mon
  547. 0/10 monc
  548. 1/ 5 paxos
  549. 0/ 5 tp
  550. 1/ 5 auth
  551. 1/ 5 crypto
  552. 1/ 1 finisher
  553. 1/ 5 heartbeatmap
  554. 1/ 5 perfcounter
  555. 1/ 5 rgw
  556. 1/10 civetweb
  557. 1/ 5 javaclient
  558. 1/ 5 asok
  559. 1/ 1 throttle
  560. 0/ 0 refs
  561. -2/-2 (syslog threshold)
  562. -1/-1 (stderr threshold)
  563. max_recent 10000
  564. max_new 1000
  565. log_file /var/log/ceph/ceph-mds.node02d.log
  566. --- end dump of recent events ---
  567. 2015-01-15 13:14:23.430403 7fa9925dc800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6755
  568. 2015-01-15 13:14:23.432838 7fa9925dc800 -1 mds.-1.0 log_to_monitors {default=true}
  569. 2015-01-15 13:14:23.436147 7fa9925dc800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  570. 2015-01-15 13:14:23.436169 7fa9925dc800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  571. 2015-01-15 13:14:23.438272 7fa9925dc800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7fa9925dc800 time 2015-01-15 13:14:23.436649
  572. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  573.  
  574. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  575. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  576. 2: (Thread::join(void**)+0x41) [0x9308a1]
  577. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  578. 4: (MDS::suicide()+0x117) [0x59bc97]
  579. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  580. 6: (main()+0x94d) [0x59038d]
  581. 7: (__libc_start_main()+0xf5) [0x7fa9905e0ec5]
  582. 8: /usr/bin/ceph-mds() [0x594d87]
  583. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  584.  
  585. --- begin dump of recent events ---
  586. -70> 2015-01-15 13:14:23.428741 7fa9925dc800 5 asok(0x4a16000) register_command perfcounters_dump hook 0x49f6050
  587. -69> 2015-01-15 13:14:23.428777 7fa9925dc800 5 asok(0x4a16000) register_command 1 hook 0x49f6050
  588. -68> 2015-01-15 13:14:23.428781 7fa9925dc800 5 asok(0x4a16000) register_command perf dump hook 0x49f6050
  589. -67> 2015-01-15 13:14:23.428786 7fa9925dc800 5 asok(0x4a16000) register_command perfcounters_schema hook 0x49f6050
  590. -66> 2015-01-15 13:14:23.428800 7fa9925dc800 5 asok(0x4a16000) register_command 2 hook 0x49f6050
  591. -65> 2015-01-15 13:14:23.428802 7fa9925dc800 5 asok(0x4a16000) register_command perf schema hook 0x49f6050
  592. -64> 2015-01-15 13:14:23.428806 7fa9925dc800 5 asok(0x4a16000) register_command perf reset hook 0x49f6050
  593. -63> 2015-01-15 13:14:23.428815 7fa9925dc800 5 asok(0x4a16000) register_command config show hook 0x49f6050
  594. -62> 2015-01-15 13:14:23.428817 7fa9925dc800 5 asok(0x4a16000) register_command config set hook 0x49f6050
  595. -61> 2015-01-15 13:14:23.428819 7fa9925dc800 5 asok(0x4a16000) register_command config get hook 0x49f6050
  596. -60> 2015-01-15 13:14:23.428821 7fa9925dc800 5 asok(0x4a16000) register_command config diff hook 0x49f6050
  597. -59> 2015-01-15 13:14:23.428823 7fa9925dc800 5 asok(0x4a16000) register_command log flush hook 0x49f6050
  598. -58> 2015-01-15 13:14:23.428825 7fa9925dc800 5 asok(0x4a16000) register_command log dump hook 0x49f6050
  599. -57> 2015-01-15 13:14:23.428827 7fa9925dc800 5 asok(0x4a16000) register_command log reopen hook 0x49f6050
  600. -56> 2015-01-15 13:14:23.430403 7fa9925dc800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6755
  601. -55> 2015-01-15 13:14:23.430849 7fa9925dc800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6755 need_addr=1
  602. -54> 2015-01-15 13:14:23.432176 7fa9925dc800 5 asok(0x4a16000) init /var/run/ceph/ceph-mds.node02d.asok
  603. -53> 2015-01-15 13:14:23.432187 7fa9925dc800 5 asok(0x4a16000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  604. -52> 2015-01-15 13:14:23.432289 7fa9925dc800 5 asok(0x4a16000) register_command 0 hook 0x49f20b8
  605. -51> 2015-01-15 13:14:23.432331 7fa9925dc800 5 asok(0x4a16000) register_command version hook 0x49f20b8
  606. -50> 2015-01-15 13:14:23.432333 7fa9925dc800 5 asok(0x4a16000) register_command git_version hook 0x49f20b8
  607. -49> 2015-01-15 13:14:23.432336 7fa9925dc800 5 asok(0x4a16000) register_command help hook 0x49f60b0
  608. -48> 2015-01-15 13:14:23.432338 7fa9925dc800 5 asok(0x4a16000) register_command get_command_descriptions hook 0x49f6150
  609. -47> 2015-01-15 13:14:23.432372 7fa9925dc800 10 monclient(hunting): build_initial_monmap
  610. -46> 2015-01-15 13:14:23.432380 7fa98e1cd700 5 asok(0x4a16000) entry start
  611. -45> 2015-01-15 13:14:23.432410 7fa9925dc800 1 -- 0.0.0.0:6800/6755 messenger.start
  612. -44> 2015-01-15 13:14:23.432457 7fa9925dc800 5 adding auth protocol: cephx
  613. -43> 2015-01-15 13:14:23.432462 7fa9925dc800 5 adding auth protocol: cephx
  614. -42> 2015-01-15 13:14:23.432551 7fa9925dc800 5 asok(0x4a16000) register_command objecter_requests hook 0x49f6140
  615. -41> 2015-01-15 13:14:23.432613 7fa9925dc800 1 accepter.accepter.start
  616. -40> 2015-01-15 13:14:23.432651 7fa9925dc800 10 monclient(hunting): init
  617. -39> 2015-01-15 13:14:23.432658 7fa9925dc800 5 adding auth protocol: cephx
  618. -38> 2015-01-15 13:14:23.432659 7fa9925dc800 10 monclient(hunting): auth_supported 2 method cephx
  619. -37> 2015-01-15 13:14:23.432755 7fa9925dc800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  620. -36> 2015-01-15 13:14:23.432832 7fa9925dc800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  621. -35> 2015-01-15 13:14:23.432838 7fa9925dc800 -1 mds.-1.0 log_to_monitors {default=true}
  622. -34> 2015-01-15 13:14:23.432848 7fa9925dc800 10 monclient(hunting): _reopen_session rank -1 name
  623. -33> 2015-01-15 13:14:23.432889 7fa9925dc800 10 monclient(hunting): picked mon.noname-a con 0x4a2cf20 addr 10.0.0.2:6789/0
  624. -32> 2015-01-15 13:14:23.432905 7fa9925dc800 10 monclient(hunting): _send_mon_message to mon.noname-a at 10.0.0.2:6789/0
  625. -31> 2015-01-15 13:14:23.432909 7fa9925dc800 1 -- 0.0.0.0:6800/6755 --> 10.0.0.2:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x4a12000 con 0x4a2cf20
  626. -30> 2015-01-15 13:14:23.432919 7fa9925dc800 10 monclient(hunting): renew_subs
  627. -29> 2015-01-15 13:14:23.433954 7fa9925da700 1 -- 10.0.0.6:6800/6755 learned my addr 10.0.0.6:6800/6755
  628. -28> 2015-01-15 13:14:23.435265 7fa98c9ca700 1 -- 10.0.0.6:6800/6755 <== mon.0 10.0.0.2:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x4a25400 con 0x4a2cf20
  629. -27> 2015-01-15 13:14:23.435312 7fa98c9ca700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  630. -26> 2015-01-15 13:14:23.435328 7fa98c9ca700 10 monclient(hunting): got monmap 2, mon.noname-a is now rank -1
  631. -25> 2015-01-15 13:14:23.435330 7fa98c9ca700 10 monclient(hunting): dump:
  632. epoch 2
  633. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  634. last_changed 2015-01-15 11:34:08.529401
  635. created 0.000000
  636. 0: 10.0.0.2:6789/0 mon.node01
  637. 1: 10.0.0.3:6789/0 mon.node02a
  638. 2: 10.0.0.4:6789/0 mon.node02b
  639. 3: 10.0.0.5:6789/0 mon.node02c
  640. 4: 10.0.0.6:6789/0 mon.node02d
  641.  
  642. -24> 2015-01-15 13:14:23.435366 7fa98c9ca700 1 -- 10.0.0.6:6800/6755 <== mon.0 10.0.0.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2715500775 0 0) 0x4a12480 con 0x4a2cf20
  643. -23> 2015-01-15 13:14:23.435390 7fa98c9ca700 10 monclient(hunting): my global_id is 4420
  644. -22> 2015-01-15 13:14:23.435436 7fa98c9ca700 10 monclient(hunting): _send_mon_message to mon.node01 at 10.0.0.2:6789/0
  645. -21> 2015-01-15 13:14:23.435446 7fa98c9ca700 1 -- 10.0.0.6:6800/6755 --> 10.0.0.2:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x4a126c0 con 0x4a2cf20
  646. -20> 2015-01-15 13:14:23.436048 7fa98c9ca700 1 -- 10.0.0.6:6800/6755 <== mon.0 10.0.0.2:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (2950516492 0 0) 0x4a126c0 con 0x4a2cf20
  647. -19> 2015-01-15 13:14:23.436084 7fa98c9ca700 1 monclient(hunting): found mon.node01
  648. -18> 2015-01-15 13:14:23.436147 7fa9925dc800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  649. -17> 2015-01-15 13:14:23.436169 7fa9925dc800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  650. -16> 2015-01-15 13:14:23.436174 7fa9925dc800 5 mds.-1.log shutdown
  651. -15> 2015-01-15 13:14:23.436339 7fa9925dc800 5 asok(0x4a16000) unregister_command status ENOENT
  652. -14> 2015-01-15 13:14:23.436352 7fa9925dc800 5 asok(0x4a16000) unregister_command dump_ops_in_flight ENOENT
  653. -13> 2015-01-15 13:14:23.436355 7fa9925dc800 5 asok(0x4a16000) unregister_command ops ENOENT
  654. -12> 2015-01-15 13:14:23.436357 7fa9925dc800 5 asok(0x4a16000) unregister_command dump_historic_ops ENOENT
  655. -11> 2015-01-15 13:14:23.436358 7fa9925dc800 5 asok(0x4a16000) unregister_command scrub_path ENOENT
  656. -10> 2015-01-15 13:14:23.436360 7fa9925dc800 5 asok(0x4a16000) unregister_command flush_path ENOENT
  657. -9> 2015-01-15 13:14:23.436361 7fa9925dc800 5 asok(0x4a16000) unregister_command session evict ENOENT
  658. -8> 2015-01-15 13:14:23.436362 7fa9925dc800 5 asok(0x4a16000) unregister_command session ls ENOENT
  659. -7> 2015-01-15 13:14:23.436364 7fa9925dc800 5 asok(0x4a16000) unregister_command flush journal ENOENT
  660. -6> 2015-01-15 13:14:23.436365 7fa9925dc800 5 asok(0x4a16000) unregister_command force_readonly ENOENT
  661. -5> 2015-01-15 13:14:23.436373 7fa9925dc800 5 asok(0x4a16000) unregister_command objecter_requests
  662. -4> 2015-01-15 13:14:23.436431 7fa9925dc800 10 monclient: shutdown
  663. -3> 2015-01-15 13:14:23.436599 7fa9925dc800 1 -- 10.0.0.6:6800/6755 mark_down 0x4a2cf20 -- 0x4aa2000
  664. -2> 2015-01-15 13:14:23.436645 7fa9899c4700 2 -- 10.0.0.6:6800/6755 >> 10.0.0.2:6789/0 pipe(0x4aa2000 sd=8 :51968 s=4 pgs=113 cs=1 l=1 c=0x4a2cf20).reader couldn't read tag, (0) Success
  665. -1> 2015-01-15 13:14:23.436690 7fa9899c4700 2 -- 10.0.0.6:6800/6755 >> 10.0.0.2:6789/0 pipe(0x4aa2000 sd=8 :51968 s=4 pgs=113 cs=1 l=1 c=0x4a2cf20).fault (0) Success
  666. 0> 2015-01-15 13:14:23.438272 7fa9925dc800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7fa9925dc800 time 2015-01-15 13:14:23.436649
  667. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  668.  
  669. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  670. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  671. 2: (Thread::join(void**)+0x41) [0x9308a1]
  672. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  673. 4: (MDS::suicide()+0x117) [0x59bc97]
  674. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  675. 6: (main()+0x94d) [0x59038d]
  676. 7: (__libc_start_main()+0xf5) [0x7fa9905e0ec5]
  677. 8: /usr/bin/ceph-mds() [0x594d87]
  678. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  679.  
  680. --- logging levels ---
  681. 0/ 5 none
  682. 0/ 1 lockdep
  683. 0/ 1 context
  684. 1/ 1 crush
  685. 1/ 5 mds
  686. 1/ 5 mds_balancer
  687. 1/ 5 mds_locker
  688. 1/ 5 mds_log
  689. 1/ 5 mds_log_expire
  690. 1/ 5 mds_migrator
  691. 0/ 1 buffer
  692. 0/ 1 timer
  693. 0/ 1 filer
  694. 0/ 1 striper
  695. 0/ 1 objecter
  696. 0/ 5 rados
  697. 0/ 5 rbd
  698. 0/ 5 rbd_replay
  699. 0/ 5 journaler
  700. 0/ 5 objectcacher
  701. 0/ 5 client
  702. 0/ 5 osd
  703. 0/ 5 optracker
  704. 0/ 5 objclass
  705. 1/ 3 filestore
  706. 1/ 3 keyvaluestore
  707. 1/ 3 journal
  708. 0/ 5 ms
  709. 1/ 5 mon
  710. 0/10 monc
  711. 1/ 5 paxos
  712. 0/ 5 tp
  713. 1/ 5 auth
  714. 1/ 5 crypto
  715. 1/ 1 finisher
  716. 1/ 5 heartbeatmap
  717. 1/ 5 perfcounter
  718. 1/ 5 rgw
  719. 1/10 civetweb
  720. 1/ 5 javaclient
  721. 1/ 5 asok
  722. 1/ 1 throttle
  723. 0/ 0 refs
  724. -2/-2 (syslog threshold)
  725. -1/-1 (stderr threshold)
  726. max_recent 10000
  727. max_new 1000
  728. log_file /var/log/ceph/ceph-mds.node02d.log
  729. --- end dump of recent events ---
  730. 2015-01-15 13:14:23.441184 7fa9925dc800 -1 *** Caught signal (Aborted) **
  731. in thread 7fa9925dc800
  732.  
  733. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  734. 1: /usr/bin/ceph-mds() [0x85721a]
  735. 2: (()+0x10340) [0x7fa991f53340]
  736. 3: (gsignal()+0x39) [0x7fa9905f5cc9]
  737. 4: (abort()+0x148) [0x7fa9905f90d8]
  738. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa990f016b5]
  739. 6: (()+0x5e836) [0x7fa990eff836]
  740. 7: (()+0x5e863) [0x7fa990eff863]
  741. 8: (()+0x5eaa2) [0x7fa990effaa2]
  742. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  743. 10: (Thread::join(void**)+0x41) [0x9308a1]
  744. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  745. 12: (MDS::suicide()+0x117) [0x59bc97]
  746. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  747. 14: (main()+0x94d) [0x59038d]
  748. 15: (__libc_start_main()+0xf5) [0x7fa9905e0ec5]
  749. 16: /usr/bin/ceph-mds() [0x594d87]
  750. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  751.  
  752. --- begin dump of recent events ---
  753. 0> 2015-01-15 13:14:23.441184 7fa9925dc800 -1 *** Caught signal (Aborted) **
  754. in thread 7fa9925dc800
  755.  
  756. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  757. 1: /usr/bin/ceph-mds() [0x85721a]
  758. 2: (()+0x10340) [0x7fa991f53340]
  759. 3: (gsignal()+0x39) [0x7fa9905f5cc9]
  760. 4: (abort()+0x148) [0x7fa9905f90d8]
  761. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa990f016b5]
  762. 6: (()+0x5e836) [0x7fa990eff836]
  763. 7: (()+0x5e863) [0x7fa990eff863]
  764. 8: (()+0x5eaa2) [0x7fa990effaa2]
  765. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  766. 10: (Thread::join(void**)+0x41) [0x9308a1]
  767. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  768. 12: (MDS::suicide()+0x117) [0x59bc97]
  769. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  770. 14: (main()+0x94d) [0x59038d]
  771. 15: (__libc_start_main()+0xf5) [0x7fa9905e0ec5]
  772. 16: /usr/bin/ceph-mds() [0x594d87]
  773. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  774.  
  775. --- logging levels ---
  776. 0/ 5 none
  777. 0/ 1 lockdep
  778. 0/ 1 context
  779. 1/ 1 crush
  780. 1/ 5 mds
  781. 1/ 5 mds_balancer
  782. 1/ 5 mds_locker
  783. 1/ 5 mds_log
  784. 1/ 5 mds_log_expire
  785. 1/ 5 mds_migrator
  786. 0/ 1 buffer
  787. 0/ 1 timer
  788. 0/ 1 filer
  789. 0/ 1 striper
  790. 0/ 1 objecter
  791. 0/ 5 rados
  792. 0/ 5 rbd
  793. 0/ 5 rbd_replay
  794. 0/ 5 journaler
  795. 0/ 5 objectcacher
  796. 0/ 5 client
  797. 0/ 5 osd
  798. 0/ 5 optracker
  799. 0/ 5 objclass
  800. 1/ 3 filestore
  801. 1/ 3 keyvaluestore
  802. 1/ 3 journal
  803. 0/ 5 ms
  804. 1/ 5 mon
  805. 0/10 monc
  806. 1/ 5 paxos
  807. 0/ 5 tp
  808. 1/ 5 auth
  809. 1/ 5 crypto
  810. 1/ 1 finisher
  811. 1/ 5 heartbeatmap
  812. 1/ 5 perfcounter
  813. 1/ 5 rgw
  814. 1/10 civetweb
  815. 1/ 5 javaclient
  816. 1/ 5 asok
  817. 1/ 1 throttle
  818. 0/ 0 refs
  819. -2/-2 (syslog threshold)
  820. -1/-1 (stderr threshold)
  821. max_recent 10000
  822. max_new 1000
  823. log_file /var/log/ceph/ceph-mds.node02d.log
  824. --- end dump of recent events ---
  825. 2015-01-15 13:14:23.453807 7f9ce4a2b800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6772
  826. 2015-01-15 13:14:23.456453 7f9ce4a2b800 -1 mds.-1.0 log_to_monitors {default=true}
  827. 2015-01-15 13:14:23.460652 7f9ce4a2b800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  828. 2015-01-15 13:14:23.460676 7f9ce4a2b800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  829. 2015-01-15 13:14:23.463020 7f9ce4a2b800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f9ce4a2b800 time 2015-01-15 13:14:23.461400
  830. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  831.  
  832. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  833. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  834. 2: (Thread::join(void**)+0x41) [0x9308a1]
  835. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  836. 4: (MDS::suicide()+0x117) [0x59bc97]
  837. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  838. 6: (main()+0x94d) [0x59038d]
  839. 7: (__libc_start_main()+0xf5) [0x7f9ce2a2fec5]
  840. 8: /usr/bin/ceph-mds() [0x594d87]
  841. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  842.  
  843. --- begin dump of recent events ---
  844. -70> 2015-01-15 13:14:23.452158 7f9ce4a2b800 5 asok(0x5374000) register_command perfcounters_dump hook 0x5354050
  845. -69> 2015-01-15 13:14:23.452191 7f9ce4a2b800 5 asok(0x5374000) register_command 1 hook 0x5354050
  846. -68> 2015-01-15 13:14:23.452195 7f9ce4a2b800 5 asok(0x5374000) register_command perf dump hook 0x5354050
  847. -67> 2015-01-15 13:14:23.452202 7f9ce4a2b800 5 asok(0x5374000) register_command perfcounters_schema hook 0x5354050
  848. -66> 2015-01-15 13:14:23.452212 7f9ce4a2b800 5 asok(0x5374000) register_command 2 hook 0x5354050
  849. -65> 2015-01-15 13:14:23.452215 7f9ce4a2b800 5 asok(0x5374000) register_command perf schema hook 0x5354050
  850. -64> 2015-01-15 13:14:23.452217 7f9ce4a2b800 5 asok(0x5374000) register_command perf reset hook 0x5354050
  851. -63> 2015-01-15 13:14:23.452220 7f9ce4a2b800 5 asok(0x5374000) register_command config show hook 0x5354050
  852. -62> 2015-01-15 13:14:23.452222 7f9ce4a2b800 5 asok(0x5374000) register_command config set hook 0x5354050
  853. -61> 2015-01-15 13:14:23.452231 7f9ce4a2b800 5 asok(0x5374000) register_command config get hook 0x5354050
  854. -60> 2015-01-15 13:14:23.452233 7f9ce4a2b800 5 asok(0x5374000) register_command config diff hook 0x5354050
  855. -59> 2015-01-15 13:14:23.452235 7f9ce4a2b800 5 asok(0x5374000) register_command log flush hook 0x5354050
  856. -58> 2015-01-15 13:14:23.452237 7f9ce4a2b800 5 asok(0x5374000) register_command log dump hook 0x5354050
  857. -57> 2015-01-15 13:14:23.452240 7f9ce4a2b800 5 asok(0x5374000) register_command log reopen hook 0x5354050
  858. -56> 2015-01-15 13:14:23.453807 7f9ce4a2b800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6772
  859. -55> 2015-01-15 13:14:23.454224 7f9ce4a2b800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6772 need_addr=1
  860. -54> 2015-01-15 13:14:23.455574 7f9ce4a2b800 5 asok(0x5374000) init /var/run/ceph/ceph-mds.node02d.asok
  861. -53> 2015-01-15 13:14:23.455585 7f9ce4a2b800 5 asok(0x5374000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  862. -52> 2015-01-15 13:14:23.455686 7f9ce4a2b800 5 asok(0x5374000) register_command 0 hook 0x53500b8
  863. -51> 2015-01-15 13:14:23.455745 7f9ce4a2b800 5 asok(0x5374000) register_command version hook 0x53500b8
  864. -50> 2015-01-15 13:14:23.455748 7f9ce4a2b800 5 asok(0x5374000) register_command git_version hook 0x53500b8
  865. -49> 2015-01-15 13:14:23.455750 7f9ce4a2b800 5 asok(0x5374000) register_command help hook 0x53540b0
  866. -48> 2015-01-15 13:14:23.455753 7f9ce4a2b800 5 asok(0x5374000) register_command get_command_descriptions hook 0x5354150
  867. -47> 2015-01-15 13:14:23.455787 7f9ce4a2b800 10 monclient(hunting): build_initial_monmap
  868. -46> 2015-01-15 13:14:23.455789 7f9ce061c700 5 asok(0x5374000) entry start
  869. -45> 2015-01-15 13:14:23.455827 7f9ce4a2b800 1 -- 0.0.0.0:6800/6772 messenger.start
  870. -44> 2015-01-15 13:14:23.455862 7f9ce4a2b800 5 adding auth protocol: cephx
  871. -43> 2015-01-15 13:14:23.455866 7f9ce4a2b800 5 adding auth protocol: cephx
  872. -42> 2015-01-15 13:14:23.455939 7f9ce4a2b800 5 asok(0x5374000) register_command objecter_requests hook 0x5354140
  873. -41> 2015-01-15 13:14:23.456006 7f9ce4a2b800 1 accepter.accepter.start
  874. -40> 2015-01-15 13:14:23.456214 7f9ce4a2b800 10 monclient(hunting): init
  875. -39> 2015-01-15 13:14:23.456230 7f9ce4a2b800 5 adding auth protocol: cephx
  876. -38> 2015-01-15 13:14:23.456233 7f9ce4a2b800 10 monclient(hunting): auth_supported 2 method cephx
  877. -37> 2015-01-15 13:14:23.456366 7f9ce4a2b800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  878. -36> 2015-01-15 13:14:23.456447 7f9ce4a2b800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  879. -35> 2015-01-15 13:14:23.456453 7f9ce4a2b800 -1 mds.-1.0 log_to_monitors {default=true}
  880. -34> 2015-01-15 13:14:23.456463 7f9ce4a2b800 10 monclient(hunting): _reopen_session rank -1 name
  881. -33> 2015-01-15 13:14:23.456507 7f9ce4a2b800 10 monclient(hunting): picked mon.noname-d con 0x538af20 addr 10.0.0.5:6789/0
  882. -32> 2015-01-15 13:14:23.456524 7f9ce4a2b800 10 monclient(hunting): _send_mon_message to mon.noname-d at 10.0.0.5:6789/0
  883. -31> 2015-01-15 13:14:23.456528 7f9ce4a2b800 1 -- 0.0.0.0:6800/6772 --> 10.0.0.5:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x5370000 con 0x538af20
  884. -30> 2015-01-15 13:14:23.456538 7f9ce4a2b800 10 monclient(hunting): renew_subs
  885. -29> 2015-01-15 13:14:23.457701 7f9ce4a29700 1 -- 10.0.0.6:6800/6772 learned my addr 10.0.0.6:6800/6772
  886. -28> 2015-01-15 13:14:23.459399 7f9cdee19700 1 -- 10.0.0.6:6800/6772 <== mon.3 10.0.0.5:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x5383400 con 0x538af20
  887. -27> 2015-01-15 13:14:23.459440 7f9cdee19700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  888. -26> 2015-01-15 13:14:23.459456 7f9cdee19700 10 monclient(hunting): got monmap 2, mon.noname-d is now rank -1
  889. -25> 2015-01-15 13:14:23.459458 7f9cdee19700 10 monclient(hunting): dump:
  890. epoch 2
  891. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  892. last_changed 2015-01-15 11:34:08.529401
  893. created 0.000000
  894. 0: 10.0.0.2:6789/0 mon.node01
  895. 1: 10.0.0.3:6789/0 mon.node02a
  896. 2: 10.0.0.4:6789/0 mon.node02b
  897. 3: 10.0.0.5:6789/0 mon.node02c
  898. 4: 10.0.0.6:6789/0 mon.node02d
  899.  
  900. -24> 2015-01-15 13:14:23.459494 7f9cdee19700 1 -- 10.0.0.6:6800/6772 <== mon.3 10.0.0.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2581403001 0 0) 0x5370480 con 0x538af20
  901. -23> 2015-01-15 13:14:23.459519 7f9cdee19700 10 monclient(hunting): my global_id is 4253
  902. -22> 2015-01-15 13:14:23.459556 7f9cdee19700 10 monclient(hunting): _send_mon_message to mon.node02c at 10.0.0.5:6789/0
  903. -21> 2015-01-15 13:14:23.459563 7f9cdee19700 1 -- 10.0.0.6:6800/6772 --> 10.0.0.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x53706c0 con 0x538af20
  904. -20> 2015-01-15 13:14:23.460568 7f9cdee19700 1 -- 10.0.0.6:6800/6772 <== mon.3 10.0.0.5:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (1278566255 0 0) 0x53706c0 con 0x538af20
  905. -19> 2015-01-15 13:14:23.460603 7f9cdee19700 1 monclient(hunting): found mon.node02c
  906. -18> 2015-01-15 13:14:23.460652 7f9ce4a2b800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  907. -17> 2015-01-15 13:14:23.460676 7f9ce4a2b800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  908. -16> 2015-01-15 13:14:23.460681 7f9ce4a2b800 5 mds.-1.log shutdown
  909. -15> 2015-01-15 13:14:23.460866 7f9ce4a2b800 5 asok(0x5374000) unregister_command status ENOENT
  910. -14> 2015-01-15 13:14:23.460878 7f9ce4a2b800 5 asok(0x5374000) unregister_command dump_ops_in_flight ENOENT
  911. -13> 2015-01-15 13:14:23.460882 7f9ce4a2b800 5 asok(0x5374000) unregister_command ops ENOENT
  912. -12> 2015-01-15 13:14:23.460884 7f9ce4a2b800 5 asok(0x5374000) unregister_command dump_historic_ops ENOENT
  913. -11> 2015-01-15 13:14:23.460885 7f9ce4a2b800 5 asok(0x5374000) unregister_command scrub_path ENOENT
  914. -10> 2015-01-15 13:14:23.460887 7f9ce4a2b800 5 asok(0x5374000) unregister_command flush_path ENOENT
  915. -9> 2015-01-15 13:14:23.460888 7f9ce4a2b800 5 asok(0x5374000) unregister_command session evict ENOENT
  916. -8> 2015-01-15 13:14:23.460890 7f9ce4a2b800 5 asok(0x5374000) unregister_command session ls ENOENT
  917. -7> 2015-01-15 13:14:23.460891 7f9ce4a2b800 5 asok(0x5374000) unregister_command flush journal ENOENT
  918. -6> 2015-01-15 13:14:23.460893 7f9ce4a2b800 5 asok(0x5374000) unregister_command force_readonly ENOENT
  919. -5> 2015-01-15 13:14:23.460901 7f9ce4a2b800 5 asok(0x5374000) unregister_command objecter_requests
  920. -4> 2015-01-15 13:14:23.461044 7f9ce4a2b800 10 monclient: shutdown
  921. -3> 2015-01-15 13:14:23.461336 7f9ce4a2b800 1 -- 10.0.0.6:6800/6772 mark_down 0x538af20 -- 0x5400000
  922. -2> 2015-01-15 13:14:23.461477 7f9cdbe13700 2 -- 10.0.0.6:6800/6772 >> 10.0.0.5:6789/0 pipe(0x5400000 sd=8 :37088 s=4 pgs=517 cs=1 l=1 c=0x538af20).reader couldn't read tag, (0) Success
  923. -1> 2015-01-15 13:14:23.461524 7f9cdbe13700 2 -- 10.0.0.6:6800/6772 >> 10.0.0.5:6789/0 pipe(0x5400000 sd=8 :37088 s=4 pgs=517 cs=1 l=1 c=0x538af20).fault (0) Success
  924. 0> 2015-01-15 13:14:23.463020 7f9ce4a2b800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f9ce4a2b800 time 2015-01-15 13:14:23.461400
  925. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  926.  
  927. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  928. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  929. 2: (Thread::join(void**)+0x41) [0x9308a1]
  930. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  931. 4: (MDS::suicide()+0x117) [0x59bc97]
  932. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  933. 6: (main()+0x94d) [0x59038d]
  934. 7: (__libc_start_main()+0xf5) [0x7f9ce2a2fec5]
  935. 8: /usr/bin/ceph-mds() [0x594d87]
  936. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  937.  
  938. --- logging levels ---
  939. 0/ 5 none
  940. 0/ 1 lockdep
  941. 0/ 1 context
  942. 1/ 1 crush
  943. 1/ 5 mds
  944. 1/ 5 mds_balancer
  945. 1/ 5 mds_locker
  946. 1/ 5 mds_log
  947. 1/ 5 mds_log_expire
  948. 1/ 5 mds_migrator
  949. 0/ 1 buffer
  950. 0/ 1 timer
  951. 0/ 1 filer
  952. 0/ 1 striper
  953. 0/ 1 objecter
  954. 0/ 5 rados
  955. 0/ 5 rbd
  956. 0/ 5 rbd_replay
  957. 0/ 5 journaler
  958. 0/ 5 objectcacher
  959. 0/ 5 client
  960. 0/ 5 osd
  961. 0/ 5 optracker
  962. 0/ 5 objclass
  963. 1/ 3 filestore
  964. 1/ 3 keyvaluestore
  965. 1/ 3 journal
  966. 0/ 5 ms
  967. 1/ 5 mon
  968. 0/10 monc
  969. 1/ 5 paxos
  970. 0/ 5 tp
  971. 1/ 5 auth
  972. 1/ 5 crypto
  973. 1/ 1 finisher
  974. 1/ 5 heartbeatmap
  975. 1/ 5 perfcounter
  976. 1/ 5 rgw
  977. 1/10 civetweb
  978. 1/ 5 javaclient
  979. 1/ 5 asok
  980. 1/ 1 throttle
  981. 0/ 0 refs
  982. -2/-2 (syslog threshold)
  983. -1/-1 (stderr threshold)
  984. max_recent 10000
  985. max_new 1000
  986. log_file /var/log/ceph/ceph-mds.node02d.log
  987. --- end dump of recent events ---
  988. 2015-01-15 13:14:23.465888 7f9ce4a2b800 -1 *** Caught signal (Aborted) **
  989. in thread 7f9ce4a2b800
  990.  
  991. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  992. 1: /usr/bin/ceph-mds() [0x85721a]
  993. 2: (()+0x10340) [0x7f9ce43a2340]
  994. 3: (gsignal()+0x39) [0x7f9ce2a44cc9]
  995. 4: (abort()+0x148) [0x7f9ce2a480d8]
  996. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9ce33506b5]
  997. 6: (()+0x5e836) [0x7f9ce334e836]
  998. 7: (()+0x5e863) [0x7f9ce334e863]
  999. 8: (()+0x5eaa2) [0x7f9ce334eaa2]
  1000. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1001. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1002. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1003. 12: (MDS::suicide()+0x117) [0x59bc97]
  1004. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1005. 14: (main()+0x94d) [0x59038d]
  1006. 15: (__libc_start_main()+0xf5) [0x7f9ce2a2fec5]
  1007. 16: /usr/bin/ceph-mds() [0x594d87]
  1008. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1009.  
  1010. --- begin dump of recent events ---
  1011. 0> 2015-01-15 13:14:23.465888 7f9ce4a2b800 -1 *** Caught signal (Aborted) **
  1012. in thread 7f9ce4a2b800
  1013.  
  1014. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1015. 1: /usr/bin/ceph-mds() [0x85721a]
  1016. 2: (()+0x10340) [0x7f9ce43a2340]
  1017. 3: (gsignal()+0x39) [0x7f9ce2a44cc9]
  1018. 4: (abort()+0x148) [0x7f9ce2a480d8]
  1019. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9ce33506b5]
  1020. 6: (()+0x5e836) [0x7f9ce334e836]
  1021. 7: (()+0x5e863) [0x7f9ce334e863]
  1022. 8: (()+0x5eaa2) [0x7f9ce334eaa2]
  1023. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1024. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1025. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1026. 12: (MDS::suicide()+0x117) [0x59bc97]
  1027. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1028. 14: (main()+0x94d) [0x59038d]
  1029. 15: (__libc_start_main()+0xf5) [0x7f9ce2a2fec5]
  1030. 16: /usr/bin/ceph-mds() [0x594d87]
  1031. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1032.  
  1033. --- logging levels ---
  1034. 0/ 5 none
  1035. 0/ 1 lockdep
  1036. 0/ 1 context
  1037. 1/ 1 crush
  1038. 1/ 5 mds
  1039. 1/ 5 mds_balancer
  1040. 1/ 5 mds_locker
  1041. 1/ 5 mds_log
  1042. 1/ 5 mds_log_expire
  1043. 1/ 5 mds_migrator
  1044. 0/ 1 buffer
  1045. 0/ 1 timer
  1046. 0/ 1 filer
  1047. 0/ 1 striper
  1048. 0/ 1 objecter
  1049. 0/ 5 rados
  1050. 0/ 5 rbd
  1051. 0/ 5 rbd_replay
  1052. 0/ 5 journaler
  1053. 0/ 5 objectcacher
  1054. 0/ 5 client
  1055. 0/ 5 osd
  1056. 0/ 5 optracker
  1057. 0/ 5 objclass
  1058. 1/ 3 filestore
  1059. 1/ 3 keyvaluestore
  1060. 1/ 3 journal
  1061. 0/ 5 ms
  1062. 1/ 5 mon
  1063. 0/10 monc
  1064. 1/ 5 paxos
  1065. 0/ 5 tp
  1066. 1/ 5 auth
  1067. 1/ 5 crypto
  1068. 1/ 1 finisher
  1069. 1/ 5 heartbeatmap
  1070. 1/ 5 perfcounter
  1071. 1/ 5 rgw
  1072. 1/10 civetweb
  1073. 1/ 5 javaclient
  1074. 1/ 5 asok
  1075. 1/ 1 throttle
  1076. 0/ 0 refs
  1077. -2/-2 (syslog threshold)
  1078. -1/-1 (stderr threshold)
  1079. max_recent 10000
  1080. max_new 1000
  1081. log_file /var/log/ceph/ceph-mds.node02d.log
  1082. --- end dump of recent events ---
  1083. 2015-01-15 13:14:23.477588 7f3adcbcf800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6789
  1084. 2015-01-15 13:14:23.480030 7f3adcbcf800 -1 mds.-1.0 log_to_monitors {default=true}
  1085. 2015-01-15 13:14:23.482725 7f3adcbcf800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  1086. 2015-01-15 13:14:23.482754 7f3adcbcf800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  1087. 2015-01-15 13:14:23.484809 7f3adcbcf800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f3adcbcf800 time 2015-01-15 13:14:23.483258
  1088. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  1089.  
  1090. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1091. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  1092. 2: (Thread::join(void**)+0x41) [0x9308a1]
  1093. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1094. 4: (MDS::suicide()+0x117) [0x59bc97]
  1095. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1096. 6: (main()+0x94d) [0x59038d]
  1097. 7: (__libc_start_main()+0xf5) [0x7f3adabd3ec5]
  1098. 8: /usr/bin/ceph-mds() [0x594d87]
  1099. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1100.  
  1101. --- begin dump of recent events ---
  1102. -70> 2015-01-15 13:14:23.475997 7f3adcbcf800 5 asok(0x3a94000) register_command perfcounters_dump hook 0x3a74050
  1103. -69> 2015-01-15 13:14:23.476028 7f3adcbcf800 5 asok(0x3a94000) register_command 1 hook 0x3a74050
  1104. -68> 2015-01-15 13:14:23.476031 7f3adcbcf800 5 asok(0x3a94000) register_command perf dump hook 0x3a74050
  1105. -67> 2015-01-15 13:14:23.476037 7f3adcbcf800 5 asok(0x3a94000) register_command perfcounters_schema hook 0x3a74050
  1106. -66> 2015-01-15 13:14:23.476041 7f3adcbcf800 5 asok(0x3a94000) register_command 2 hook 0x3a74050
  1107. -65> 2015-01-15 13:14:23.476046 7f3adcbcf800 5 asok(0x3a94000) register_command perf schema hook 0x3a74050
  1108. -64> 2015-01-15 13:14:23.476053 7f3adcbcf800 5 asok(0x3a94000) register_command perf reset hook 0x3a74050
  1109. -63> 2015-01-15 13:14:23.476056 7f3adcbcf800 5 asok(0x3a94000) register_command config show hook 0x3a74050
  1110. -62> 2015-01-15 13:14:23.476058 7f3adcbcf800 5 asok(0x3a94000) register_command config set hook 0x3a74050
  1111. -61> 2015-01-15 13:14:23.476060 7f3adcbcf800 5 asok(0x3a94000) register_command config get hook 0x3a74050
  1112. -60> 2015-01-15 13:14:23.476066 7f3adcbcf800 5 asok(0x3a94000) register_command config diff hook 0x3a74050
  1113. -59> 2015-01-15 13:14:23.476068 7f3adcbcf800 5 asok(0x3a94000) register_command log flush hook 0x3a74050
  1114. -58> 2015-01-15 13:14:23.476070 7f3adcbcf800 5 asok(0x3a94000) register_command log dump hook 0x3a74050
  1115. -57> 2015-01-15 13:14:23.476073 7f3adcbcf800 5 asok(0x3a94000) register_command log reopen hook 0x3a74050
  1116. -56> 2015-01-15 13:14:23.477588 7f3adcbcf800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6789
  1117. -55> 2015-01-15 13:14:23.477998 7f3adcbcf800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6789 need_addr=1
  1118. -54> 2015-01-15 13:14:23.479288 7f3adcbcf800 5 asok(0x3a94000) init /var/run/ceph/ceph-mds.node02d.asok
  1119. -53> 2015-01-15 13:14:23.479299 7f3adcbcf800 5 asok(0x3a94000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  1120. -52> 2015-01-15 13:14:23.479423 7f3adcbcf800 5 asok(0x3a94000) register_command 0 hook 0x3a700b8
  1121. -51> 2015-01-15 13:14:23.479467 7f3adcbcf800 5 asok(0x3a94000) register_command version hook 0x3a700b8
  1122. -50> 2015-01-15 13:14:23.479470 7f3adcbcf800 5 asok(0x3a94000) register_command git_version hook 0x3a700b8
  1123. -49> 2015-01-15 13:14:23.479472 7f3adcbcf800 5 asok(0x3a94000) register_command help hook 0x3a740b0
  1124. -48> 2015-01-15 13:14:23.479475 7f3adcbcf800 5 asok(0x3a94000) register_command get_command_descriptions hook 0x3a74150
  1125. -47> 2015-01-15 13:14:23.479508 7f3adcbcf800 10 monclient(hunting): build_initial_monmap
  1126. -46> 2015-01-15 13:14:23.479542 7f3adcbcf800 1 -- 0.0.0.0:6800/6789 messenger.start
  1127. -45> 2015-01-15 13:14:23.479587 7f3adcbcf800 5 adding auth protocol: cephx
  1128. -44> 2015-01-15 13:14:23.479592 7f3adcbcf800 5 adding auth protocol: cephx
  1129. -43> 2015-01-15 13:14:23.479580 7f3ad87c0700 5 asok(0x3a94000) entry start
  1130. -42> 2015-01-15 13:14:23.479679 7f3adcbcf800 5 asok(0x3a94000) register_command objecter_requests hook 0x3a74140
  1131. -41> 2015-01-15 13:14:23.479808 7f3adcbcf800 1 accepter.accepter.start
  1132. -40> 2015-01-15 13:14:23.479831 7f3adcbcf800 10 monclient(hunting): init
  1133. -39> 2015-01-15 13:14:23.479838 7f3adcbcf800 5 adding auth protocol: cephx
  1134. -38> 2015-01-15 13:14:23.479840 7f3adcbcf800 10 monclient(hunting): auth_supported 2 method cephx
  1135. -37> 2015-01-15 13:14:23.479928 7f3adcbcf800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  1136. -36> 2015-01-15 13:14:23.480025 7f3adcbcf800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  1137. -35> 2015-01-15 13:14:23.480030 7f3adcbcf800 -1 mds.-1.0 log_to_monitors {default=true}
  1138. -34> 2015-01-15 13:14:23.480061 7f3adcbcf800 10 monclient(hunting): _reopen_session rank -1 name
  1139. -33> 2015-01-15 13:14:23.480099 7f3adcbcf800 10 monclient(hunting): picked mon.noname-c con 0x3aaaf20 addr 10.0.0.4:6789/0
  1140. -32> 2015-01-15 13:14:23.480115 7f3adcbcf800 10 monclient(hunting): _send_mon_message to mon.noname-c at 10.0.0.4:6789/0
  1141. -31> 2015-01-15 13:14:23.480167 7f3adcbcf800 1 -- 0.0.0.0:6800/6789 --> 10.0.0.4:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x3a90000 con 0x3aaaf20
  1142. -30> 2015-01-15 13:14:23.480177 7f3adcbcf800 10 monclient(hunting): renew_subs
  1143. -29> 2015-01-15 13:14:23.480788 7f3adcbcd700 1 -- 10.0.0.6:6800/6789 learned my addr 10.0.0.6:6800/6789
  1144. -28> 2015-01-15 13:14:23.481772 7f3ad6fbd700 1 -- 10.0.0.6:6800/6789 <== mon.2 10.0.0.4:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x3aa3400 con 0x3aaaf20
  1145. -27> 2015-01-15 13:14:23.481820 7f3ad6fbd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  1146. -26> 2015-01-15 13:14:23.481839 7f3ad6fbd700 10 monclient(hunting): got monmap 2, mon.noname-c is now rank -1
  1147. -25> 2015-01-15 13:14:23.481842 7f3ad6fbd700 10 monclient(hunting): dump:
  1148. epoch 2
  1149. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  1150. last_changed 2015-01-15 11:34:08.529401
  1151. created 0.000000
  1152. 0: 10.0.0.2:6789/0 mon.node01
  1153. 1: 10.0.0.3:6789/0 mon.node02a
  1154. 2: 10.0.0.4:6789/0 mon.node02b
  1155. 3: 10.0.0.5:6789/0 mon.node02c
  1156. 4: 10.0.0.6:6789/0 mon.node02d
  1157.  
  1158. -24> 2015-01-15 13:14:23.481884 7f3ad6fbd700 1 -- 10.0.0.6:6800/6789 <== mon.2 10.0.0.4:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2213122035 0 0) 0x3a90480 con 0x3aaaf20
  1159. -23> 2015-01-15 13:14:23.481915 7f3ad6fbd700 10 monclient(hunting): my global_id is 4242
  1160. -22> 2015-01-15 13:14:23.481962 7f3ad6fbd700 10 monclient(hunting): _send_mon_message to mon.node02b at 10.0.0.4:6789/0
  1161. -21> 2015-01-15 13:14:23.481969 7f3ad6fbd700 1 -- 10.0.0.6:6800/6789 --> 10.0.0.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3a906c0 con 0x3aaaf20
  1162. -20> 2015-01-15 13:14:23.482607 7f3ad6fbd700 1 -- 10.0.0.6:6800/6789 <== mon.2 10.0.0.4:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (271367248 0 0) 0x3a906c0 con 0x3aaaf20
  1163. -19> 2015-01-15 13:14:23.482641 7f3ad6fbd700 1 monclient(hunting): found mon.node02b
  1164. -18> 2015-01-15 13:14:23.482725 7f3adcbcf800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  1165. -17> 2015-01-15 13:14:23.482754 7f3adcbcf800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  1166. -16> 2015-01-15 13:14:23.482761 7f3adcbcf800 5 mds.-1.log shutdown
  1167. -15> 2015-01-15 13:14:23.482873 7f3adcbcf800 5 asok(0x3a94000) unregister_command status ENOENT
  1168. -14> 2015-01-15 13:14:23.482880 7f3adcbcf800 5 asok(0x3a94000) unregister_command dump_ops_in_flight ENOENT
  1169. -13> 2015-01-15 13:14:23.482883 7f3adcbcf800 5 asok(0x3a94000) unregister_command ops ENOENT
  1170. -12> 2015-01-15 13:14:23.482885 7f3adcbcf800 5 asok(0x3a94000) unregister_command dump_historic_ops ENOENT
  1171. -11> 2015-01-15 13:14:23.482886 7f3adcbcf800 5 asok(0x3a94000) unregister_command scrub_path ENOENT
  1172. -10> 2015-01-15 13:14:23.482888 7f3adcbcf800 5 asok(0x3a94000) unregister_command flush_path ENOENT
  1173. -9> 2015-01-15 13:14:23.482889 7f3adcbcf800 5 asok(0x3a94000) unregister_command session evict ENOENT
  1174. -8> 2015-01-15 13:14:23.482893 7f3adcbcf800 5 asok(0x3a94000) unregister_command session ls ENOENT
  1175. -7> 2015-01-15 13:14:23.482899 7f3adcbcf800 5 asok(0x3a94000) unregister_command flush journal ENOENT
  1176. -6> 2015-01-15 13:14:23.482904 7f3adcbcf800 5 asok(0x3a94000) unregister_command force_readonly ENOENT
  1177. -5> 2015-01-15 13:14:23.482914 7f3adcbcf800 5 asok(0x3a94000) unregister_command objecter_requests
  1178. -4> 2015-01-15 13:14:23.482967 7f3adcbcf800 10 monclient: shutdown
  1179. -3> 2015-01-15 13:14:23.483207 7f3adcbcf800 1 -- 10.0.0.6:6800/6789 mark_down 0x3aaaf20 -- 0x3b20000
  1180. -2> 2015-01-15 13:14:23.483251 7f3ad3fb7700 2 -- 10.0.0.6:6800/6789 >> 10.0.0.4:6789/0 pipe(0x3b20000 sd=8 :50907 s=4 pgs=923 cs=1 l=1 c=0x3aaaf20).reader couldn't read tag, (0) Success
  1181. -1> 2015-01-15 13:14:23.483291 7f3ad3fb7700 2 -- 10.0.0.6:6800/6789 >> 10.0.0.4:6789/0 pipe(0x3b20000 sd=8 :50907 s=4 pgs=923 cs=1 l=1 c=0x3aaaf20).fault (0) Success
  1182. 0> 2015-01-15 13:14:23.484809 7f3adcbcf800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f3adcbcf800 time 2015-01-15 13:14:23.483258
  1183. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  1184.  
  1185. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1186. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  1187. 2: (Thread::join(void**)+0x41) [0x9308a1]
  1188. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1189. 4: (MDS::suicide()+0x117) [0x59bc97]
  1190. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1191. 6: (main()+0x94d) [0x59038d]
  1192. 7: (__libc_start_main()+0xf5) [0x7f3adabd3ec5]
  1193. 8: /usr/bin/ceph-mds() [0x594d87]
  1194. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1195.  
  1196. --- logging levels ---
  1197. 0/ 5 none
  1198. 0/ 1 lockdep
  1199. 0/ 1 context
  1200. 1/ 1 crush
  1201. 1/ 5 mds
  1202. 1/ 5 mds_balancer
  1203. 1/ 5 mds_locker
  1204. 1/ 5 mds_log
  1205. 1/ 5 mds_log_expire
  1206. 1/ 5 mds_migrator
  1207. 0/ 1 buffer
  1208. 0/ 1 timer
  1209. 0/ 1 filer
  1210. 0/ 1 striper
  1211. 0/ 1 objecter
  1212. 0/ 5 rados
  1213. 0/ 5 rbd
  1214. 0/ 5 rbd_replay
  1215. 0/ 5 journaler
  1216. 0/ 5 objectcacher
  1217. 0/ 5 client
  1218. 0/ 5 osd
  1219. 0/ 5 optracker
  1220. 0/ 5 objclass
  1221. 1/ 3 filestore
  1222. 1/ 3 keyvaluestore
  1223. 1/ 3 journal
  1224. 0/ 5 ms
  1225. 1/ 5 mon
  1226. 0/10 monc
  1227. 1/ 5 paxos
  1228. 0/ 5 tp
  1229. 1/ 5 auth
  1230. 1/ 5 crypto
  1231. 1/ 1 finisher
  1232. 1/ 5 heartbeatmap
  1233. 1/ 5 perfcounter
  1234. 1/ 5 rgw
  1235. 1/10 civetweb
  1236. 1/ 5 javaclient
  1237. 1/ 5 asok
  1238. 1/ 1 throttle
  1239. 0/ 0 refs
  1240. -2/-2 (syslog threshold)
  1241. -1/-1 (stderr threshold)
  1242. max_recent 10000
  1243. max_new 1000
  1244. log_file /var/log/ceph/ceph-mds.node02d.log
  1245. --- end dump of recent events ---
  1246. 2015-01-15 13:14:23.487354 7f3adcbcf800 -1 *** Caught signal (Aborted) **
  1247. in thread 7f3adcbcf800
  1248.  
  1249. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1250. 1: /usr/bin/ceph-mds() [0x85721a]
  1251. 2: (()+0x10340) [0x7f3adc546340]
  1252. 3: (gsignal()+0x39) [0x7f3adabe8cc9]
  1253. 4: (abort()+0x148) [0x7f3adabec0d8]
  1254. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3adb4f46b5]
  1255. 6: (()+0x5e836) [0x7f3adb4f2836]
  1256. 7: (()+0x5e863) [0x7f3adb4f2863]
  1257. 8: (()+0x5eaa2) [0x7f3adb4f2aa2]
  1258. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1259. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1260. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1261. 12: (MDS::suicide()+0x117) [0x59bc97]
  1262. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1263. 14: (main()+0x94d) [0x59038d]
  1264. 15: (__libc_start_main()+0xf5) [0x7f3adabd3ec5]
  1265. 16: /usr/bin/ceph-mds() [0x594d87]
  1266. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1267.  
  1268. --- begin dump of recent events ---
  1269. 0> 2015-01-15 13:14:23.487354 7f3adcbcf800 -1 *** Caught signal (Aborted) **
  1270. in thread 7f3adcbcf800
  1271.  
  1272. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1273. 1: /usr/bin/ceph-mds() [0x85721a]
  1274. 2: (()+0x10340) [0x7f3adc546340]
  1275. 3: (gsignal()+0x39) [0x7f3adabe8cc9]
  1276. 4: (abort()+0x148) [0x7f3adabec0d8]
  1277. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3adb4f46b5]
  1278. 6: (()+0x5e836) [0x7f3adb4f2836]
  1279. 7: (()+0x5e863) [0x7f3adb4f2863]
  1280. 8: (()+0x5eaa2) [0x7f3adb4f2aa2]
  1281. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1282. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1283. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1284. 12: (MDS::suicide()+0x117) [0x59bc97]
  1285. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1286. 14: (main()+0x94d) [0x59038d]
  1287. 15: (__libc_start_main()+0xf5) [0x7f3adabd3ec5]
  1288. 16: /usr/bin/ceph-mds() [0x594d87]
  1289. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1290.  
  1291. --- logging levels ---
  1292. 0/ 5 none
  1293. 0/ 1 lockdep
  1294. 0/ 1 context
  1295. 1/ 1 crush
  1296. 1/ 5 mds
  1297. 1/ 5 mds_balancer
  1298. 1/ 5 mds_locker
  1299. 1/ 5 mds_log
  1300. 1/ 5 mds_log_expire
  1301. 1/ 5 mds_migrator
  1302. 0/ 1 buffer
  1303. 0/ 1 timer
  1304. 0/ 1 filer
  1305. 0/ 1 striper
  1306. 0/ 1 objecter
  1307. 0/ 5 rados
  1308. 0/ 5 rbd
  1309. 0/ 5 rbd_replay
  1310. 0/ 5 journaler
  1311. 0/ 5 objectcacher
  1312. 0/ 5 client
  1313. 0/ 5 osd
  1314. 0/ 5 optracker
  1315. 0/ 5 objclass
  1316. 1/ 3 filestore
  1317. 1/ 3 keyvaluestore
  1318. 1/ 3 journal
  1319. 0/ 5 ms
  1320. 1/ 5 mon
  1321. 0/10 monc
  1322. 1/ 5 paxos
  1323. 0/ 5 tp
  1324. 1/ 5 auth
  1325. 1/ 5 crypto
  1326. 1/ 1 finisher
  1327. 1/ 5 heartbeatmap
  1328. 1/ 5 perfcounter
  1329. 1/ 5 rgw
  1330. 1/10 civetweb
  1331. 1/ 5 javaclient
  1332. 1/ 5 asok
  1333. 1/ 1 throttle
  1334. 0/ 0 refs
  1335. -2/-2 (syslog threshold)
  1336. -1/-1 (stderr threshold)
  1337. max_recent 10000
  1338. max_new 1000
  1339. log_file /var/log/ceph/ceph-mds.node02d.log
  1340. --- end dump of recent events ---
  1341. 2015-01-15 13:14:23.499905 7f40d5e78800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6806
  1342. 2015-01-15 13:14:23.502261 7f40d5e78800 -1 mds.-1.0 log_to_monitors {default=true}
  1343. 2015-01-15 13:14:23.505466 7f40d5e78800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  1344. 2015-01-15 13:14:23.505490 7f40d5e78800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  1345. 2015-01-15 13:14:23.507642 7f40d5e78800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f40d5e78800 time 2015-01-15 13:14:23.506050
  1346. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  1347.  
  1348. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1349. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  1350. 2: (Thread::join(void**)+0x41) [0x9308a1]
  1351. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1352. 4: (MDS::suicide()+0x117) [0x59bc97]
  1353. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1354. 6: (main()+0x94d) [0x59038d]
  1355. 7: (__libc_start_main()+0xf5) [0x7f40d3e7cec5]
  1356. 8: /usr/bin/ceph-mds() [0x594d87]
  1357. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1358.  
  1359. --- begin dump of recent events ---
  1360. -70> 2015-01-15 13:14:23.498171 7f40d5e78800 5 asok(0x3be2000) register_command perfcounters_dump hook 0x3bc2050
  1361. -69> 2015-01-15 13:14:23.498207 7f40d5e78800 5 asok(0x3be2000) register_command 1 hook 0x3bc2050
  1362. -68> 2015-01-15 13:14:23.498210 7f40d5e78800 5 asok(0x3be2000) register_command perf dump hook 0x3bc2050
  1363. -67> 2015-01-15 13:14:23.498216 7f40d5e78800 5 asok(0x3be2000) register_command perfcounters_schema hook 0x3bc2050
  1364. -66> 2015-01-15 13:14:23.498222 7f40d5e78800 5 asok(0x3be2000) register_command 2 hook 0x3bc2050
  1365. -65> 2015-01-15 13:14:23.498238 7f40d5e78800 5 asok(0x3be2000) register_command perf schema hook 0x3bc2050
  1366. -64> 2015-01-15 13:14:23.498241 7f40d5e78800 5 asok(0x3be2000) register_command perf reset hook 0x3bc2050
  1367. -63> 2015-01-15 13:14:23.498243 7f40d5e78800 5 asok(0x3be2000) register_command config show hook 0x3bc2050
  1368. -62> 2015-01-15 13:14:23.498246 7f40d5e78800 5 asok(0x3be2000) register_command config set hook 0x3bc2050
  1369. -61> 2015-01-15 13:14:23.498263 7f40d5e78800 5 asok(0x3be2000) register_command config get hook 0x3bc2050
  1370. -60> 2015-01-15 13:14:23.498265 7f40d5e78800 5 asok(0x3be2000) register_command config diff hook 0x3bc2050
  1371. -59> 2015-01-15 13:14:23.498267 7f40d5e78800 5 asok(0x3be2000) register_command log flush hook 0x3bc2050
  1372. -58> 2015-01-15 13:14:23.498270 7f40d5e78800 5 asok(0x3be2000) register_command log dump hook 0x3bc2050
  1373. -57> 2015-01-15 13:14:23.498272 7f40d5e78800 5 asok(0x3be2000) register_command log reopen hook 0x3bc2050
  1374. -56> 2015-01-15 13:14:23.499905 7f40d5e78800 0 ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc), process ceph-mds, pid 6806
  1375. -55> 2015-01-15 13:14:23.500367 7f40d5e78800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/6806 need_addr=1
  1376. -54> 2015-01-15 13:14:23.501618 7f40d5e78800 5 asok(0x3be2000) init /var/run/ceph/ceph-mds.node02d.asok
  1377. -53> 2015-01-15 13:14:23.501629 7f40d5e78800 5 asok(0x3be2000) bind_and_listen /var/run/ceph/ceph-mds.node02d.asok
  1378. -52> 2015-01-15 13:14:23.501723 7f40d5e78800 5 asok(0x3be2000) register_command 0 hook 0x3bbe0b8
  1379. -51> 2015-01-15 13:14:23.501763 7f40d5e78800 5 asok(0x3be2000) register_command version hook 0x3bbe0b8
  1380. -50> 2015-01-15 13:14:23.501766 7f40d5e78800 5 asok(0x3be2000) register_command git_version hook 0x3bbe0b8
  1381. -49> 2015-01-15 13:14:23.501768 7f40d5e78800 5 asok(0x3be2000) register_command help hook 0x3bc20b0
  1382. -48> 2015-01-15 13:14:23.501771 7f40d5e78800 5 asok(0x3be2000) register_command get_command_descriptions hook 0x3bc2150
  1383. -47> 2015-01-15 13:14:23.501804 7f40d5e78800 10 monclient(hunting): build_initial_monmap
  1384. -46> 2015-01-15 13:14:23.501805 7f40d1a69700 5 asok(0x3be2000) entry start
  1385. -45> 2015-01-15 13:14:23.501844 7f40d5e78800 1 -- 0.0.0.0:6800/6806 messenger.start
  1386. -44> 2015-01-15 13:14:23.501889 7f40d5e78800 5 adding auth protocol: cephx
  1387. -43> 2015-01-15 13:14:23.501894 7f40d5e78800 5 adding auth protocol: cephx
  1388. -42> 2015-01-15 13:14:23.501984 7f40d5e78800 5 asok(0x3be2000) register_command objecter_requests hook 0x3bc2140
  1389. -41> 2015-01-15 13:14:23.502045 7f40d5e78800 1 accepter.accepter.start
  1390. -40> 2015-01-15 13:14:23.502068 7f40d5e78800 10 monclient(hunting): init
  1391. -39> 2015-01-15 13:14:23.502076 7f40d5e78800 5 adding auth protocol: cephx
  1392. -38> 2015-01-15 13:14:23.502078 7f40d5e78800 10 monclient(hunting): auth_supported 2 method cephx
  1393. -37> 2015-01-15 13:14:23.502174 7f40d5e78800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-node02d/keyring
  1394. -36> 2015-01-15 13:14:23.502255 7f40d5e78800 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
  1395. -35> 2015-01-15 13:14:23.502261 7f40d5e78800 -1 mds.-1.0 log_to_monitors {default=true}
  1396. -34> 2015-01-15 13:14:23.502272 7f40d5e78800 10 monclient(hunting): _reopen_session rank -1 name
  1397. -33> 2015-01-15 13:14:23.502313 7f40d5e78800 10 monclient(hunting): picked mon.noname-a con 0x3bf8f20 addr 10.0.0.2:6789/0
  1398. -32> 2015-01-15 13:14:23.502327 7f40d5e78800 10 monclient(hunting): _send_mon_message to mon.noname-a at 10.0.0.2:6789/0
  1399. -31> 2015-01-15 13:14:23.502331 7f40d5e78800 1 -- 0.0.0.0:6800/6806 --> 10.0.0.2:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0x3bde000 con 0x3bf8f20
  1400. -30> 2015-01-15 13:14:23.502342 7f40d5e78800 10 monclient(hunting): renew_subs
  1401. -29> 2015-01-15 13:14:23.503308 7f40d5e76700 1 -- 10.0.0.6:6800/6806 learned my addr 10.0.0.6:6800/6806
  1402. -28> 2015-01-15 13:14:23.504578 7f40d0266700 1 -- 10.0.0.6:6800/6806 <== mon.0 10.0.0.2:6789/0 1 ==== mon_map magic: 0 v1 ==== 784+0+0 (776308501 0 0) 0x3bf1400 con 0x3bf8f20
  1403. -27> 2015-01-15 13:14:23.504621 7f40d0266700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  1404. -26> 2015-01-15 13:14:23.504661 7f40d0266700 10 monclient(hunting): got monmap 2, mon.noname-a is now rank -1
  1405. -25> 2015-01-15 13:14:23.504666 7f40d0266700 10 monclient(hunting): dump:
  1406. epoch 2
  1407. fsid 200f03c0-e22f-47a6-8302-fb9091519dc0
  1408. last_changed 2015-01-15 11:34:08.529401
  1409. created 0.000000
  1410. 0: 10.0.0.2:6789/0 mon.node01
  1411. 1: 10.0.0.3:6789/0 mon.node02a
  1412. 2: 10.0.0.4:6789/0 mon.node02b
  1413. 3: 10.0.0.5:6789/0 mon.node02c
  1414. 4: 10.0.0.6:6789/0 mon.node02d
  1415.  
  1416. -24> 2015-01-15 13:14:23.504704 7f40d0266700 1 -- 10.0.0.6:6800/6806 <== mon.0 10.0.0.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (715770217 0 0) 0x3bde480 con 0x3bf8f20
  1417. -23> 2015-01-15 13:14:23.504730 7f40d0266700 10 monclient(hunting): my global_id is 4425
  1418. -22> 2015-01-15 13:14:23.504774 7f40d0266700 10 monclient(hunting): _send_mon_message to mon.node01 at 10.0.0.2:6789/0
  1419. -21> 2015-01-15 13:14:23.504782 7f40d0266700 1 -- 10.0.0.6:6800/6806 --> 10.0.0.2:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3bde6c0 con 0x3bf8f20
  1420. -20> 2015-01-15 13:14:23.505387 7f40d0266700 1 -- 10.0.0.6:6800/6806 <== mon.0 10.0.0.2:6789/0 3 ==== auth_reply(proto 2 -1 (1) Operation not permitted) v1 ==== 24+0+0 (305185086 0 0) 0x3bde6c0 con 0x3bf8f20
  1421. -19> 2015-01-15 13:14:23.505413 7f40d0266700 1 monclient(hunting): found mon.node01
  1422. -18> 2015-01-15 13:14:23.505466 7f40d5e78800 -1 mds.-1.0 ERROR: failed to authenticate: (1) Operation not permitted
  1423. -17> 2015-01-15 13:14:23.505490 7f40d5e78800 1 mds.-1.0 suicide. wanted down:dne, now up:boot
  1424. -16> 2015-01-15 13:14:23.505496 7f40d5e78800 5 mds.-1.log shutdown
  1425. -15> 2015-01-15 13:14:23.505643 7f40d5e78800 5 asok(0x3be2000) unregister_command status ENOENT
  1426. -14> 2015-01-15 13:14:23.505658 7f40d5e78800 5 asok(0x3be2000) unregister_command dump_ops_in_flight ENOENT
  1427. -13> 2015-01-15 13:14:23.505662 7f40d5e78800 5 asok(0x3be2000) unregister_command ops ENOENT
  1428. -12> 2015-01-15 13:14:23.505664 7f40d5e78800 5 asok(0x3be2000) unregister_command dump_historic_ops ENOENT
  1429. -11> 2015-01-15 13:14:23.505665 7f40d5e78800 5 asok(0x3be2000) unregister_command scrub_path ENOENT
  1430. -10> 2015-01-15 13:14:23.505666 7f40d5e78800 5 asok(0x3be2000) unregister_command flush_path ENOENT
  1431. -9> 2015-01-15 13:14:23.505668 7f40d5e78800 5 asok(0x3be2000) unregister_command session evict ENOENT
  1432. -8> 2015-01-15 13:14:23.505669 7f40d5e78800 5 asok(0x3be2000) unregister_command session ls ENOENT
  1433. -7> 2015-01-15 13:14:23.505671 7f40d5e78800 5 asok(0x3be2000) unregister_command flush journal ENOENT
  1434. -6> 2015-01-15 13:14:23.505672 7f40d5e78800 5 asok(0x3be2000) unregister_command force_readonly ENOENT
  1435. -5> 2015-01-15 13:14:23.505680 7f40d5e78800 5 asok(0x3be2000) unregister_command objecter_requests
  1436. -4> 2015-01-15 13:14:23.505832 7f40d5e78800 10 monclient: shutdown
  1437. -3> 2015-01-15 13:14:23.506003 7f40d5e78800 1 -- 10.0.0.6:6800/6806 mark_down 0x3bf8f20 -- 0x3c6e000
  1438. -2> 2015-01-15 13:14:23.506126 7f40cd260700 2 -- 10.0.0.6:6800/6806 >> 10.0.0.2:6789/0 pipe(0x3c6e000 sd=8 :51971 s=4 pgs=114 cs=1 l=1 c=0x3bf8f20).reader couldn't read tag, (0) Success
  1439. -1> 2015-01-15 13:14:23.506177 7f40cd260700 2 -- 10.0.0.6:6800/6806 >> 10.0.0.2:6789/0 pipe(0x3c6e000 sd=8 :51971 s=4 pgs=114 cs=1 l=1 c=0x3bf8f20).fault (0) Success
  1440. 0> 2015-01-15 13:14:23.507642 7f40d5e78800 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f40d5e78800 time 2015-01-15 13:14:23.506050
  1441. common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
  1442.  
  1443. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1444. 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x94427b]
  1445. 2: (Thread::join(void**)+0x41) [0x9308a1]
  1446. 3: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1447. 4: (MDS::suicide()+0x117) [0x59bc97]
  1448. 5: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1449. 6: (main()+0x94d) [0x59038d]
  1450. 7: (__libc_start_main()+0xf5) [0x7f40d3e7cec5]
  1451. 8: /usr/bin/ceph-mds() [0x594d87]
  1452. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1453.  
  1454. --- logging levels ---
  1455. 0/ 5 none
  1456. 0/ 1 lockdep
  1457. 0/ 1 context
  1458. 1/ 1 crush
  1459. 1/ 5 mds
  1460. 1/ 5 mds_balancer
  1461. 1/ 5 mds_locker
  1462. 1/ 5 mds_log
  1463. 1/ 5 mds_log_expire
  1464. 1/ 5 mds_migrator
  1465. 0/ 1 buffer
  1466. 0/ 1 timer
  1467. 0/ 1 filer
  1468. 0/ 1 striper
  1469. 0/ 1 objecter
  1470. 0/ 5 rados
  1471. 0/ 5 rbd
  1472. 0/ 5 rbd_replay
  1473. 0/ 5 journaler
  1474. 0/ 5 objectcacher
  1475. 0/ 5 client
  1476. 0/ 5 osd
  1477. 0/ 5 optracker
  1478. 0/ 5 objclass
  1479. 1/ 3 filestore
  1480. 1/ 3 keyvaluestore
  1481. 1/ 3 journal
  1482. 0/ 5 ms
  1483. 1/ 5 mon
  1484. 0/10 monc
  1485. 1/ 5 paxos
  1486. 0/ 5 tp
  1487. 1/ 5 auth
  1488. 1/ 5 crypto
  1489. 1/ 1 finisher
  1490. 1/ 5 heartbeatmap
  1491. 1/ 5 perfcounter
  1492. 1/ 5 rgw
  1493. 1/10 civetweb
  1494. 1/ 5 javaclient
  1495. 1/ 5 asok
  1496. 1/ 1 throttle
  1497. 0/ 0 refs
  1498. -2/-2 (syslog threshold)
  1499. -1/-1 (stderr threshold)
  1500. max_recent 10000
  1501. max_new 1000
  1502. log_file /var/log/ceph/ceph-mds.node02d.log
  1503. --- end dump of recent events ---
  1504. 2015-01-15 13:14:23.510468 7f40d5e78800 -1 *** Caught signal (Aborted) **
  1505. in thread 7f40d5e78800
  1506.  
  1507. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1508. 1: /usr/bin/ceph-mds() [0x85721a]
  1509. 2: (()+0x10340) [0x7f40d57ef340]
  1510. 3: (gsignal()+0x39) [0x7f40d3e91cc9]
  1511. 4: (abort()+0x148) [0x7f40d3e950d8]
  1512. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f40d479d6b5]
  1513. 6: (()+0x5e836) [0x7f40d479b836]
  1514. 7: (()+0x5e863) [0x7f40d479b863]
  1515. 8: (()+0x5eaa2) [0x7f40d479baa2]
  1516. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1517. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1518. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1519. 12: (MDS::suicide()+0x117) [0x59bc97]
  1520. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1521. 14: (main()+0x94d) [0x59038d]
  1522. 15: (__libc_start_main()+0xf5) [0x7f40d3e7cec5]
  1523. 16: /usr/bin/ceph-mds() [0x594d87]
  1524. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1525.  
  1526. --- begin dump of recent events ---
  1527. 0> 2015-01-15 13:14:23.510468 7f40d5e78800 -1 *** Caught signal (Aborted) **
  1528. in thread 7f40d5e78800
  1529.  
  1530. ceph version 0.91-348-gd4a6447 (d4a64474e53ce7c9472feac530ca94ccf616fbcc)
  1531. 1: /usr/bin/ceph-mds() [0x85721a]
  1532. 2: (()+0x10340) [0x7f40d57ef340]
  1533. 3: (gsignal()+0x39) [0x7f40d3e91cc9]
  1534. 4: (abort()+0x148) [0x7f40d3e950d8]
  1535. 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f40d479d6b5]
  1536. 6: (()+0x5e836) [0x7f40d479b836]
  1537. 7: (()+0x5e863) [0x7f40d479b863]
  1538. 8: (()+0x5eaa2) [0x7f40d479baa2]
  1539. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x944468]
  1540. 10: (Thread::join(void**)+0x41) [0x9308a1]
  1541. 11: (MDS::ProgressThread::shutdown()+0x74) [0x59bb64]
  1542. 12: (MDS::suicide()+0x117) [0x59bc97]
  1543. 13: (MDS::init(MDSMap::DaemonState)+0x2013) [0x5acc03]
  1544. 14: (main()+0x94d) [0x59038d]
  1545. 15: (__libc_start_main()+0xf5) [0x7f40d3e7cec5]
  1546. 16: /usr/bin/ceph-mds() [0x594d87]
  1547. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1548.  
  1549. --- logging levels ---
  1550. 0/ 5 none
  1551. 0/ 1 lockdep
  1552. 0/ 1 context
  1553. 1/ 1 crush
  1554. 1/ 5 mds
  1555. 1/ 5 mds_balancer
  1556. 1/ 5 mds_locker
  1557. 1/ 5 mds_log
  1558. 1/ 5 mds_log_expire
  1559. 1/ 5 mds_migrator
  1560. 0/ 1 buffer
  1561. 0/ 1 timer
  1562. 0/ 1 filer
  1563. 0/ 1 striper
  1564. 0/ 1 objecter
  1565. 0/ 5 rados
  1566. 0/ 5 rbd
  1567. 0/ 5 rbd_replay
  1568. 0/ 5 journaler
  1569. 0/ 5 objectcacher
  1570. 0/ 5 client
  1571. 0/ 5 osd
  1572. 0/ 5 optracker
  1573. 0/ 5 objclass
  1574. 1/ 3 filestore
  1575. 1/ 3 keyvaluestore
  1576. 1/ 3 journal
  1577. 0/ 5 ms
  1578. 1/ 5 mon
  1579. 0/10 monc
  1580. 1/ 5 paxos
  1581. 0/ 5 tp
  1582. 1/ 5 auth
  1583. 1/ 5 crypto
  1584. 1/ 1 finisher
  1585. 1/ 5 heartbeatmap
  1586. 1/ 5 perfcounter
  1587. 1/ 5 rgw
  1588. 1/10 civetweb
  1589. 1/ 5 javaclient
  1590. 1/ 5 asok
  1591. 1/ 1 throttle
  1592. 0/ 0 refs
  1593. -2/-2 (syslog threshold)
  1594. -1/-1 (stderr threshold)
  1595. max_recent 10000
  1596. max_new 1000
  1597. log_file /var/log/ceph/ceph-mds.node02d.log
  1598. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!