API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Sep 17th, 2013
72
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 99.51 KB | None | 0 0
raw download clone embed print report
  1. -25> 2013-09-17 16:36:45.459804 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.33 [2001:620:0:6::106]:6812/4773 2 ==== osd_op_reply(32 200.000081a5 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1163729470 0 1239038857) 0x1dcc600 con 0x1509ce0
  2. -24> 2013-09-17 16:36:45.459859 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.95:34 200.000081a7 [read 0~4194304] 1.6dd25d51 e65913) v4 -- ?+0 0x21f7b00 con 0x14e3760
  3. -23> 2013-09-17 16:36:45.470301 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.5 [2001:620:0:6::104]:6806/25076 2 ==== osd_op_reply(28 200.000081a1 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1489336063 0 4024501100) 0x156ba00 con 0x157ba20
  4. -22> 2013-09-17 16:36:45.470698 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.42 [2001:620:0:6::10a]:6822/16408 2 ==== osd_op_reply(33 200.000081a6 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1041185256 0 3695802034) 0x1dcca00 con 0x157b4a0
  5. -21> 2013-09-17 16:36:45.480727 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.17 [2001:620:0:6::110]:6803/4416 3 ==== osd_op_reply(34 200.000081a7 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (995135743 0 640228953) 0x156a400 con 0x14e3760
  6. -20> 2013-09-17 16:36:45.488789 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::112]:6803/5058 -- osd_op(mds.0.95:35 200.000081a8 [read 0~4194304] 1.ef569196 e65913) v4 -- ?+0 0x21f7680 con 0x1dd0c60
  7. -19> 2013-09-17 16:36:45.516346 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.95:36 200.000081a9 [read 0~4194304] 1.9a15953a e65913) v4 -- ?+0 0x21f7200 con 0x1509080
  8. -18> 2013-09-17 16:36:45.543776 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::10e]:6824/6100 -- osd_op(mds.0.95:37 200.000081aa [read 0~4194304] 1.6c6009b4 e65913) v4 -- ?+0 0x142cd80 con 0x1509600
  9. -17> 2013-09-17 16:36:45.582413 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::10a]:6819/15996 -- osd_op(mds.0.95:38 200.000081ab [read 0~4194304] 1.2bfafcbd e65913) v4 -- ?+0 0x1debd40 con 0x1dd09a0
  10. -16> 2013-09-17 16:36:45.589920 7faa9bde0700 5 mds.0.95 ms_handle_connect on [2001:620:0:6::112]:6803/5058
  11. -15> 2013-09-17 16:36:45.615121 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::108]:6809/8955 -- osd_op(mds.0.95:39 200.000081ac [read 0~4194304] 1.2b1c95db e65913) v4 -- ?+0 0x1deb8c0 con 0x1dd06e0
  12. -14> 2013-09-17 16:36:45.617079 7faa9bde0700 5 mds.0.95 ms_handle_connect on [2001:620:0:6::10a]:6819/15996
  13. -13> 2013-09-17 16:36:45.643072 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::108]:6812/10542 -- osd_op(mds.0.95:40 200.000081ad [read 0~4194304] 1.4dc097e4 e65913) v4 -- ?+0 0x1deb440 con 0x1dd0420
  14. -12> 2013-09-17 16:36:45.670765 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.95:41 200.000081ae [read 0~4194304] 1.115a19cf e65913) v4 -- ?+0 0x1deafc0 con 0x1509ce0
  15. -11> 2013-09-17 16:36:45.698438 7faa980c6700 1 -- [2001:620:0:6::d0]:6800/14026 --> [2001:620:0:6::10a]:6807/15040 -- osd_op(mds.0.95:42 200.000081af [read 0~1206981] 1.e01c7684 e65913) v4 -- ?+0 0x1504fc0 con 0x14e2c60
  16. -10> 2013-09-17 16:36:45.710159 7faa9bde0700 5 mds.0.95 ms_handle_connect on [2001:620:0:6::108]:6809/8955
  17. -9> 2013-09-17 16:36:45.751786 7faa9bde0700 5 mds.0.95 ms_handle_connect on [2001:620:0:6::108]:6812/10542
  18. -8> 2013-09-17 16:36:45.751813 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.57 [2001:620:0:6::112]:6803/5058 1 ==== osd_op_reply(35 200.000081a8 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2702002814 0 2756327013) 0x143be00 con 0x1dd0c60
  19. -7> 2013-09-17 16:36:45.777049 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.40 [2001:620:0:6::10a]:6819/15996 1 ==== osd_op_reply(38 200.000081ab [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (544470147 0 3103833645) 0x156b600 con 0x1dd09a0
  20. -6> 2013-09-17 16:36:45.811010 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.8 [2001:620:0:6::108]:6809/8955 1 ==== osd_op_reply(39 200.000081ac [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (674303290 0 277755) 0x156b000 con 0x1dd06e0
  21. -5> 2013-09-17 16:36:45.811045 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.9 [2001:620:0:6::108]:6812/10542 1 ==== osd_op_reply(40 200.000081ad [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1970191679 0 626561037) 0x156be00 con 0x1dd0420
  22. -4> 2013-09-17 16:36:45.811068 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.38 [2001:620:0:6::10a]:6807/15040 2 ==== osd_op_reply(42 200.000081af [read 0~1206981] ondisk = 0) v4 ==== 111+0+1206981 (3639840961 0 119534278) 0x1dcc200 con 0x14e2c60
  23. -3> 2013-09-17 16:36:45.811090 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.1 [2001:620:0:6::100]:6803/17838 3 ==== osd_op_reply(36 200.000081a9 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1704818953 0 1428356768) 0x143bc00 con 0x1509080
  24. -2> 2013-09-17 16:36:45.811117 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.55 [2001:620:0:6::10e]:6824/6100 2 ==== osd_op_reply(37 200.000081aa [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2330473852 0 1683739580) 0x156a600 con 0x1509600
  25. -1> 2013-09-17 16:36:45.811140 7faa9bde0700 1 -- [2001:620:0:6::d0]:6800/14026 <== osd.33 [2001:620:0:6::106]:6812/4773 3 ==== osd_op_reply(41 200.000081ae [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2317679555 0 3604356958) 0x1dcc400 con 0x1509ce0
  26. 0> 2013-09-17 16:36:45.973856 7faa980c6700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7faa980c6700 time 2013-09-17 16:36:45.965910
  27. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  28.  
  29. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  30. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  31. 2: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  32. 3: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  33. 4: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  34. 5: (()+0x7e9a) [0x7faaa09d4e9a]
  35. 6: (clone()+0x6d) [0x7faa9f871cbd]
  36. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  37.  
  38. --- logging levels ---
  39. 0/ 5 none
  40. 0/ 1 lockdep
  41. 0/ 1 context
  42. 1/ 1 crush
  43. 1/ 5 mds
  44. 1/ 5 mds_balancer
  45. 1/ 5 mds_locker
  46. 1/ 5 mds_log
  47. 1/ 5 mds_log_expire
  48. 1/ 5 mds_migrator
  49. 0/ 1 buffer
  50. 0/ 1 timer
  51. 0/ 1 filer
  52. 0/ 1 striper
  53. 0/ 1 objecter
  54. 0/ 5 rados
  55. 0/ 5 rbd
  56. 0/ 5 journaler
  57. 0/ 5 objectcacher
  58. 0/ 5 client
  59. 0/ 5 osd
  60. 0/ 5 optracker
  61. 0/ 5 objclass
  62. 1/ 3 filestore
  63. 1/ 3 journal
  64. 0/ 5 ms
  65. 1/ 5 mon
  66. 0/10 monc
  67. 0/ 5 paxos
  68. 0/ 5 tp
  69. 1/ 5 auth
  70. 1/ 5 crypto
  71. 1/ 1 finisher
  72. 1/ 5 heartbeatmap
  73. 1/ 5 perfcounter
  74. 1/ 5 rgw
  75. 1/ 5 hadoop
  76. 1/ 5 javaclient
  77. 1/ 5 asok
  78. 1/ 1 throttle
  79. -2/-2 (syslog threshold)
  80. -1/-1 (stderr threshold)
  81. max_recent 10000
  82. max_new 1000
  83. log_file /var/log/ceph/ceph-mds.ineri.log
  84. --- end dump of recent events ---
  85. 2013-09-17 16:36:45.982010 7faa980c6700 -1 *** Caught signal (Aborted) **
  86. in thread 7faa980c6700
  87.  
  88. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  89. 1: /usr/bin/ceph-mds() [0x864f00]
  90. 2: (()+0xfcb0) [0x7faaa09dccb0]
  91. 3: (gsignal()+0x35) [0x7faa9f7b4425]
  92. 4: (abort()+0x17b) [0x7faa9f7b7b8b]
  93. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7faaa00afe2d]
  94. 6: (()+0x5ef26) [0x7faaa00adf26]
  95. 7: (()+0x5ef53) [0x7faaa00adf53]
  96. 8: (()+0x5f17e) [0x7faaa00ae17e]
  97. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cb11d]
  98. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  99. 11: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  100. 12: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  101. 13: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  102. 14: (()+0x7e9a) [0x7faaa09d4e9a]
  103. 15: (clone()+0x6d) [0x7faa9f871cbd]
  104. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  105.  
  106. --- begin dump of recent events ---
  107. 0> 2013-09-17 16:36:45.982010 7faa980c6700 -1 *** Caught signal (Aborted) **
  108. in thread 7faa980c6700
  109.  
  110. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  111. 1: /usr/bin/ceph-mds() [0x864f00]
  112. 2: (()+0xfcb0) [0x7faaa09dccb0]
  113. 3: (gsignal()+0x35) [0x7faa9f7b4425]
  114. 4: (abort()+0x17b) [0x7faa9f7b7b8b]
  115. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7faaa00afe2d]
  116. 6: (()+0x5ef26) [0x7faaa00adf26]
  117. 7: (()+0x5ef53) [0x7faaa00adf53]
  118. 8: (()+0x5f17e) [0x7faaa00ae17e]
  119. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cb11d]
  120. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  121. 11: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  122. 12: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  123. 13: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  124. 14: (()+0x7e9a) [0x7faaa09d4e9a]
  125. 15: (clone()+0x6d) [0x7faa9f871cbd]
  126. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  127.  
  128. --- logging levels ---
  129. 0/ 5 none
  130. 0/ 1 lockdep
  131. 0/ 1 context
  132. 1/ 1 crush
  133. 1/ 5 mds
  134. 1/ 5 mds_balancer
  135. 1/ 5 mds_locker
  136. 1/ 5 mds_log
  137. 1/ 5 mds_log_expire
  138. 1/ 5 mds_migrator
  139. 0/ 1 buffer
  140. 0/ 1 timer
  141. 0/ 1 filer
  142. 0/ 1 striper
  143. 0/ 1 objecter
  144. 0/ 5 rados
  145. 0/ 5 rbd
  146. 0/ 5 journaler
  147. 0/ 5 objectcacher
  148. 0/ 5 client
  149. 0/ 5 osd
  150. 0/ 5 optracker
  151. 0/ 5 objclass
  152. 1/ 3 filestore
  153. 1/ 3 journal
  154. 0/ 5 ms
  155. 1/ 5 mon
  156. 0/10 monc
  157. 0/ 5 paxos
  158. 0/ 5 tp
  159. 1/ 5 auth
  160. 1/ 5 crypto
  161. 1/ 1 finisher
  162. 1/ 5 heartbeatmap
  163. 1/ 5 perfcounter
  164. 1/ 5 rgw
  165. 1/ 5 hadoop
  166. 1/ 5 javaclient
  167. 1/ 5 asok
  168. 1/ 1 throttle
  169. -2/-2 (syslog threshold)
  170. -1/-1 (stderr threshold)
  171. max_recent 10000
  172. max_new 1000
  173. log_file /var/log/ceph/ceph-mds.ineri.log
  174. --- end dump of recent events ---
  175. 2013-09-17 16:39:14.434711 7f793d4bb780 0 ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404), process ceph-mds, pid 15023
  176. 2013-09-17 16:39:14.458029 7f7937f63700 1 mds.-1.0 handle_mds_map standby
  177. 2013-09-17 16:39:26.048078 7f7937f63700 1 mds.0.97 handle_mds_map i am now mds.0.97
  178. 2013-09-17 16:39:26.048082 7f7937f63700 1 mds.0.97 handle_mds_map state change up:standby --> up:replay
  179. 2013-09-17 16:39:26.048091 7f7937f63700 1 mds.0.97 replay_start
  180. 2013-09-17 16:39:26.048100 7f7937f63700 1 mds.0.97 recovery set is
  181. 2013-09-17 16:39:26.048103 7f7937f63700 1 mds.0.97 need osdmap epoch 65927, have 65924
  182. 2013-09-17 16:39:26.048106 7f7937f63700 1 mds.0.97 waiting for osdmap 65927 (which blacklists prior instance)
  183. 2013-09-17 16:39:26.096641 7f7937f63700 0 mds.0.cache creating system inode with ino:100
  184. 2013-09-17 16:39:26.096903 7f7937f63700 0 mds.0.cache creating system inode with ino:1
  185. 2013-09-17 16:39:27.036674 7f7934249700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f7934249700 time 2013-09-17 16:39:27.035936
  186. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  187.  
  188. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  189. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  190. 2: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  191. 3: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  192. 4: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  193. 5: (()+0x7e9a) [0x7f793cb57e9a]
  194. 6: (clone()+0x6d) [0x7f793b9f4cbd]
  195. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  196.  
  197. --- begin dump of recent events ---
  198. -240> 2013-09-17 16:39:14.433302 7f793d4bb780 5 asok(0x1a22000) register_command perfcounters_dump hook 0x1a0e010
  199. -239> 2013-09-17 16:39:14.433325 7f793d4bb780 5 asok(0x1a22000) register_command 1 hook 0x1a0e010
  200. -238> 2013-09-17 16:39:14.433327 7f793d4bb780 5 asok(0x1a22000) register_command perf dump hook 0x1a0e010
  201. -237> 2013-09-17 16:39:14.433333 7f793d4bb780 5 asok(0x1a22000) register_command perfcounters_schema hook 0x1a0e010
  202. -236> 2013-09-17 16:39:14.433334 7f793d4bb780 5 asok(0x1a22000) register_command 2 hook 0x1a0e010
  203. -235> 2013-09-17 16:39:14.433335 7f793d4bb780 5 asok(0x1a22000) register_command perf schema hook 0x1a0e010
  204. -234> 2013-09-17 16:39:14.433338 7f793d4bb780 5 asok(0x1a22000) register_command config show hook 0x1a0e010
  205. -233> 2013-09-17 16:39:14.433340 7f793d4bb780 5 asok(0x1a22000) register_command config set hook 0x1a0e010
  206. -232> 2013-09-17 16:39:14.433341 7f793d4bb780 5 asok(0x1a22000) register_command log flush hook 0x1a0e010
  207. -231> 2013-09-17 16:39:14.433342 7f793d4bb780 5 asok(0x1a22000) register_command log dump hook 0x1a0e010
  208. -230> 2013-09-17 16:39:14.433344 7f793d4bb780 5 asok(0x1a22000) register_command log reopen hook 0x1a0e010
  209. -229> 2013-09-17 16:39:14.434711 7f793d4bb780 0 ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404), process ceph-mds, pid 15023
  210. -228> 2013-09-17 16:39:14.435229 7f793d4bb780 1 accepter.accepter.bind my_inst.addr is [::]:6800/15023 need_addr=1
  211. -227> 2013-09-17 16:39:14.435910 7f793d4bb780 1 finished global_init_daemonize
  212. -226> 2013-09-17 16:39:14.438853 7f793d4bb780 5 asok(0x1a22000) init /var/run/ceph/ceph-mds.ineri.asok
  213. -225> 2013-09-17 16:39:14.438894 7f793d4bb780 5 asok(0x1a22000) bind_and_listen /var/run/ceph/ceph-mds.ineri.asok
  214. -224> 2013-09-17 16:39:14.438935 7f793d4bb780 5 asok(0x1a22000) register_command 0 hook 0x1a0c0b8
  215. -223> 2013-09-17 16:39:14.438974 7f793d4bb780 5 asok(0x1a22000) register_command version hook 0x1a0c0b8
  216. -222> 2013-09-17 16:39:14.438981 7f793d4bb780 5 asok(0x1a22000) register_command git_version hook 0x1a0c0b8
  217. -221> 2013-09-17 16:39:14.438987 7f793d4bb780 5 asok(0x1a22000) register_command help hook 0x1a0e0d0
  218. -220> 2013-09-17 16:39:14.439039 7f793d4bb780 10 monclient(hunting): build_initial_monmap
  219. -219> 2013-09-17 16:39:14.439053 7f7939766700 5 asok(0x1a22000) entry start
  220. -218> 2013-09-17 16:39:14.439199 7f793d4bb780 1 -- [::]:6800/15023 messenger.start
  221. -217> 2013-09-17 16:39:14.439330 7f793d4bb780 5 adding auth protocol: cephx
  222. -216> 2013-09-17 16:39:14.439346 7f793d4bb780 5 adding auth protocol: cephx
  223. -215> 2013-09-17 16:39:14.439471 7f793d4bb780 1 accepter.accepter.start
  224. -214> 2013-09-17 16:39:14.439502 7f793d4bb780 10 monclient(hunting): init
  225. -213> 2013-09-17 16:39:14.439515 7f793d4bb780 5 adding auth protocol: cephx
  226. -212> 2013-09-17 16:39:14.439517 7f793d4bb780 10 monclient(hunting): auth_supported 2 method cephx
  227. -211> 2013-09-17 16:39:14.439766 7f793d4bb780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ineri/keyring
  228. -210> 2013-09-17 16:39:14.439842 7f793d4bb780 10 monclient(hunting): _reopen_session
  229. -209> 2013-09-17 16:39:14.439895 7f793d4bb780 10 monclient(hunting): _pick_new_mon picked mon.h1 con 0x1a34f20 addr [2001:620:0:6::106]:6789/0
  230. -208> 2013-09-17 16:39:14.439917 7f793d4bb780 10 monclient(hunting): _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  231. -207> 2013-09-17 16:39:14.439922 7f793d4bb780 1 -- [::]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x1a20480 con 0x1a34f20
  232. -206> 2013-09-17 16:39:14.439940 7f793d4bb780 10 monclient(hunting): renew_subs
  233. -205> 2013-09-17 16:39:14.440559 7f793d4b9700 1 -- [2001:620:0:6::d0]:6800/15023 learned my addr [2001:620:0:6::d0]:6800/15023
  234. -204> 2013-09-17 16:39:14.440838 7f7937f63700 5 mds.-1.0 ms_handle_connect on [2001:620:0:6::106]:6789/0
  235. -203> 2013-09-17 16:39:14.441387 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 1 ==== mon_map v1 ==== 476+0+0 (335251837 0 0) 0x1a903c0 con 0x1a34f20
  236. -202> 2013-09-17 16:39:14.441423 7f7937f63700 10 monclient(hunting): handle_monmap mon_map v1
  237. -201> 2013-09-17 16:39:14.441444 7f7937f63700 10 monclient(hunting): got monmap 10, mon.h1 is now rank 0
  238. -200> 2013-09-17 16:39:14.441451 7f7937f63700 10 monclient(hunting): dump:
  239. epoch 10
  240. fsid 2fe1a358-7f88-4a1b-a31b-ba7501870c80
  241. last_changed 2013-09-09 10:59:40.635175
  242. created 2013-01-28 06:53:12.936035
  243. 0: [2001:620:0:6::106]:6789/0 mon.h1
  244. 1: [2001:620:0:6::10c]:6789/0 mon.h5
  245. 2: [2001:620:0:6::10e]:6789/0 mon.h3
  246.  
  247. -199> 2013-09-17 16:39:14.441507 7f7937f63700 1 monclient(hunting): found mon.h1
  248. -198> 2013-09-17 16:39:14.441522 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1431482908 0 0) 0x1a2f400 con 0x1a34f20
  249. -197> 2013-09-17 16:39:14.441563 7f7937f63700 10 monclient: my global_id is 182994
  250. -196> 2013-09-17 16:39:14.441702 7f7937f63700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  251. -195> 2013-09-17 16:39:14.441716 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1a206c0 con 0x1a34f20
  252. -194> 2013-09-17 16:39:14.442537 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (711722922 0 0) 0x1a2f800 con 0x1a34f20
  253. -193> 2013-09-17 16:39:14.442659 7f7937f63700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  254. -192> 2013-09-17 16:39:14.442672 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1a20b40 con 0x1a34f20
  255. -191> 2013-09-17 16:39:14.443551 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (1048790689 0 0) 0x1a2f600 con 0x1a34f20
  256. -190> 2013-09-17 16:39:14.443659 7f7937f63700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  257. -189> 2013-09-17 16:39:14.443673 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1a6a380 con 0x1a34f20
  258. -188> 2013-09-17 16:39:14.443740 7f7937f63700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-09-17 16:38:44.443739)
  259. -187> 2013-09-17 16:39:14.443753 7f7937f63700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  260. -186> 2013-09-17 16:39:14.443764 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1a20900 con 0x1a34f20
  261. -185> 2013-09-17 16:39:14.443792 7f793d4bb780 5 monclient: authenticate success, global_id 182994
  262. -184> 2013-09-17 16:39:14.443804 7f793d4bb780 10 monclient: wait_auth_rotating waiting (until 2013-09-17 16:39:44.443803)
  263. -183> 2013-09-17 16:39:14.444164 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 5 ==== mon_map v1 ==== 476+0+0 (335251837 0 0) 0x1a905a0 con 0x1a34f20
  264. -182> 2013-09-17 16:39:14.444182 7f7937f63700 10 monclient: handle_monmap mon_map v1
  265. -181> 2013-09-17 16:39:14.444196 7f7937f63700 10 monclient: got monmap 10, mon.h1 is now rank 0
  266. -180> 2013-09-17 16:39:14.444202 7f7937f63700 10 monclient: dump:
  267. epoch 10
  268. fsid 2fe1a358-7f88-4a1b-a31b-ba7501870c80
  269. last_changed 2013-09-09 10:59:40.635175
  270. created 2013-01-28 06:53:12.936035
  271. 0: [2001:620:0:6::106]:6789/0 mon.h1
  272. 1: [2001:620:0:6::10c]:6789/0 mon.h5
  273. 2: [2001:620:0:6::10e]:6789/0 mon.h3
  274.  
  275. -179> 2013-09-17 16:39:14.444237 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x1a6a540 con 0x1a34f20
  276. -178> 2013-09-17 16:39:14.444256 7f7937f63700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:39:14.439948 renew after 2013-09-17 16:41:44.439948
  277. -177> 2013-09-17 16:39:14.444310 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (4055521148 0 0) 0x1a2fa00 con 0x1a34f20
  278. -176> 2013-09-17 16:39:14.444375 7f7937f63700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-09-17 16:38:44.444375)
  279. -175> 2013-09-17 16:39:14.444392 7f793d4bb780 10 monclient: wait_auth_rotating done
  280. -174> 2013-09-17 16:39:14.444417 7f793d4bb780 5 asok(0x1a22000) register_command objecter_requests hook 0x1a0e100
  281. -173> 2013-09-17 16:39:14.444474 7f793d4bb780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  282. -172> 2013-09-17 16:39:14.444485 7f793d4bb780 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mdsbeacon(182994/ineri up:boot seq 1 v0) v2 -- ?+0 0x1a782c0 con 0x1a34f20
  283. -171> 2013-09-17 16:39:14.444539 7f793d4bb780 10 monclient: renew_subs
  284. -170> 2013-09-17 16:39:14.444544 7f793d4bb780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  285. -169> 2013-09-17 16:39:14.444550 7f793d4bb780 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({monmap=11+,osdmap=0}) v2 -- ?+0 0x1a6a1c0 con 0x1a34f20
  286. -168> 2013-09-17 16:39:14.444567 7f793d4bb780 10 monclient: renew_subs
  287. -167> 2013-09-17 16:39:14.444571 7f793d4bb780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  288. -166> 2013-09-17 16:39:14.444576 7f793d4bb780 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({mdsmap=0+,monmap=11+,osdmap=0}) v2 -- ?+0 0x1a6aa80 con 0x1a34f20
  289. -165> 2013-09-17 16:39:14.449914 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 8 ==== osd_map(65924..65924 src has 24657..65924) v3 ==== 42315+0+0 (585019490 0 0) 0x1a20fc0 con 0x1a34f20
  290. -164> 2013-09-17 16:39:14.450356 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x1a6a1c0 con 0x1a34f20
  291. -163> 2013-09-17 16:39:14.450378 7f7937f63700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:39:14.444543 renew after 2013-09-17 16:41:44.444543
  292. -162> 2013-09-17 16:39:14.450401 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 10 ==== mdsmap(e 709) v1 ==== 595+0+0 (2392824145 0 0) 0x1a2f800 con 0x1a34f20
  293. -161> 2013-09-17 16:39:14.450422 7f7937f63700 5 mds.-1.0 handle_mds_map epoch 709 from mon.0
  294. -160> 2013-09-17 16:39:14.450473 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 11 ==== osd_map(65924..65924 src has 24657..65924) v3 ==== 42315+0+0 (585019490 0 0) 0x1a21440 con 0x1a34f20
  295. -159> 2013-09-17 16:39:14.450499 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x1a6a380 con 0x1a34f20
  296. -158> 2013-09-17 16:39:14.450515 7f7937f63700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  297. -157> 2013-09-17 16:39:14.457969 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 13 ==== mdsmap(e 710) v1 ==== 802+0+0 (457540563 0 0) 0x1a2f400 con 0x1a34f20
  298. -156> 2013-09-17 16:39:14.457996 7f7937f63700 5 mds.-1.-1 handle_mds_map epoch 710 from mon.0
  299. -155> 2013-09-17 16:39:14.458029 7f7937f63700 1 mds.-1.0 handle_mds_map standby
  300. -154> 2013-09-17 16:39:17.439922 7f7936f61700 10 monclient: tick
  301. -153> 2013-09-17 16:39:17.439952 7f7936f61700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-09-17 16:38:47.439950)
  302. -152> 2013-09-17 16:39:17.439967 7f7936f61700 10 monclient: renew subs? (now: 2013-09-17 16:39:17.439967; renew after: 2013-09-17 16:41:44.444543) -- no
  303. -151> 2013-09-17 16:39:18.444657 7f7935e5e700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  304. -150> 2013-09-17 16:39:18.444694 7f7935e5e700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mdsbeacon(182994/ineri up:standby seq 2 v710) v2 -- ?+0 0x1a78840 con 0x1a34f20
  305. -149> 2013-09-17 16:39:18.445726 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 14 ==== mdsbeacon(182994/ineri up:standby seq 2 v710) v2 ==== 107+0+0 (491946157 0 0) 0x1a782c0 con 0x1a34f20
  306. -148> 2013-09-17 16:39:22.444844 7f7935e5e700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  307. -147> 2013-09-17 16:39:22.444875 7f7935e5e700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mdsbeacon(182994/ineri up:standby seq 3 v710) v2 -- ?+0 0x1a78b00 con 0x1a34f20
  308. -146> 2013-09-17 16:39:22.445688 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 15 ==== mdsbeacon(182994/ineri up:standby seq 3 v710) v2 ==== 107+0+0 (3582675681 0 0) 0x1a79080 con 0x1a34f20
  309. -145> 2013-09-17 16:39:26.047952 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 16 ==== mdsmap(e 711) v1 ==== 595+0+0 (168657357 0 0) 0x1a2fc00 con 0x1a34f20
  310. -144> 2013-09-17 16:39:26.048005 7f7937f63700 5 mds.-1.0 handle_mds_map epoch 711 from mon.0
  311. -143> 2013-09-17 16:39:26.048064 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 mark_down [2001:620:0:6::d1]:6800/1202 -- pipe dne
  312. -142> 2013-09-17 16:39:26.048078 7f7937f63700 1 mds.0.97 handle_mds_map i am now mds.0.97
  313. -141> 2013-09-17 16:39:26.048082 7f7937f63700 1 mds.0.97 handle_mds_map state change up:standby --> up:replay
  314. -140> 2013-09-17 16:39:26.048091 7f7937f63700 1 mds.0.97 replay_start
  315. -139> 2013-09-17 16:39:26.048100 7f7937f63700 1 mds.0.97 recovery set is
  316. -138> 2013-09-17 16:39:26.048103 7f7937f63700 1 mds.0.97 need osdmap epoch 65927, have 65924
  317. -137> 2013-09-17 16:39:26.048106 7f7937f63700 1 mds.0.97 waiting for osdmap 65927 (which blacklists prior instance)
  318. -136> 2013-09-17 16:39:26.048119 7f7937f63700 10 monclient: renew_subs
  319. -135> 2013-09-17 16:39:26.048127 7f7937f63700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  320. -134> 2013-09-17 16:39:26.048136 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({mdsmap=712+,monmap=11+,osdmap=65925}) v2 -- ?+0 0x1a6a380 con 0x1a34f20
  321. -133> 2013-09-17 16:39:26.048167 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 mark_down [2001:620:0:6::d1]:6800/1202 -- pipe dne
  322. -132> 2013-09-17 16:39:26.048180 7f7937f63700 5 mds.0.97 handle_mds_failure for myself; not doing anything
  323. -131> 2013-09-17 16:39:26.064295 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 17 ==== osd_map(65925..65927 src has 24657..65927) v3 ==== 609+0+0 (2758885855 0 0) 0x1a21200 con 0x1a34f20
  324. -130> 2013-09-17 16:39:26.064378 7f7937f63700 2 mds.0.97 boot_start 1: opening inotable
  325. -129> 2013-09-17 16:39:26.064520 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10c]:6816/5936 -- osd_op(mds.0.97:1 mds0_inotable [read 0~0] 1.b852b893 e65927) v4 -- ?+0 0x1a21680 con 0x1a35760
  326. -128> 2013-09-17 16:39:26.064543 7f7937f63700 2 mds.0.97 boot_start 1: opening sessionmap
  327. -127> 2013-09-17 16:39:26.064603 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::102]:6809/4481 -- osd_op(mds.0.97:2 mds0_sessionmap [read 0~0] 1.3270c60b e65927) v4 -- ?+0 0x1a21440 con 0x1ad6160
  328. -126> 2013-09-17 16:39:26.064616 7f7937f63700 2 mds.0.97 boot_start 1: opening anchor table
  329. -125> 2013-09-17 16:39:26.064655 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::102]:6818/5149 -- osd_op(mds.0.97:3 mds_anchortable [read 0~0] 1.a977f6a7 e65927) v4 -- ?+0 0x1a20fc0 con 0x1ad6840
  330. -124> 2013-09-17 16:39:26.064669 7f7937f63700 2 mds.0.97 boot_start 1: opening snap table
  331. -123> 2013-09-17 16:39:26.064713 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6821/5118 -- osd_op(mds.0.97:4 mds_snaptable [read 0~0] 1.d90270ad e65927) v4 -- ?+0 0x1a20d80 con 0x1ad6580
  332. -122> 2013-09-17 16:39:26.064727 7f7937f63700 2 mds.0.97 boot_start 1: opening mds log
  333. -121> 2013-09-17 16:39:26.064731 7f7937f63700 5 mds.0.log open discovering log bounds
  334. -120> 2013-09-17 16:39:26.064738 7f7937f63700 1 mds.0.journaler(ro) recover start
  335. -119> 2013-09-17 16:39:26.064742 7f7937f63700 1 mds.0.journaler(ro) read_head
  336. -118> 2013-09-17 16:39:26.064788 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10e]:6800/5420 -- osd_op(mds.0.97:5 200.00000000 [read 0~0] 1.844f3494 e65927) v4 -- ?+0 0x1a206c0 con 0x1ad6f20
  337. -117> 2013-09-17 16:39:26.064817 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 18 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x1a6ac40 con 0x1a34f20
  338. -116> 2013-09-17 16:39:26.064828 7f7937f63700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:39:26.048122 renew after 2013-09-17 16:41:56.048122
  339. -115> 2013-09-17 16:39:26.065791 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10c]:6816/5936
  340. -114> 2013-09-17 16:39:26.065848 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::106]:6821/5118
  341. -113> 2013-09-17 16:39:26.065861 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::102]:6818/5149
  342. -112> 2013-09-17 16:39:26.065872 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::102]:6809/4481
  343. -111> 2013-09-17 16:39:26.065883 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10e]:6800/5420
  344. -110> 2013-09-17 16:39:26.067137 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.23 [2001:620:0:6::102]:6809/4481 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~1938] ondisk = 0) v4 ==== 114+0+1938 (764411222 0 1042821830) 0x1a2fe00 con 0x1ad6160
  345. -109> 2013-09-17 16:39:26.074893 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.26 [2001:620:0:6::102]:6818/5149 1 ==== osd_op_reply(3 mds_anchortable [read 0~2058] ondisk = 0) v4 ==== 114+0+2058 (232249137 0 1136844878) 0x1afc000 con 0x1ad6840
  346. -108> 2013-09-17 16:39:26.075120 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.47 [2001:620:0:6::10e]:6800/5420 1 ==== osd_op_reply(5 200.00000000 [read 0~84] ondisk = 0) v4 ==== 111+0+84 (3816703057 0 1876858853) 0x1afc200 con 0x1ad6f20
  347. -107> 2013-09-17 16:39:26.075157 7f7937f63700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 139120869376, expire 139120871988, write 139247862597). probing for end of log (from 139247862597)...
  348. -106> 2013-09-17 16:39:26.075168 7f7937f63700 1 mds.0.journaler(ro) probing for end of the log
  349. -105> 2013-09-17 16:39:26.075273 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6807/15040 -- osd_op(mds.0.97:6 200.000081af [stat] 1.e01c7684 rwordered e65927) v4 -- ?+0 0x1af38c0 con 0x1ad6c60
  350. -104> 2013-09-17 16:39:26.075449 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6816/16018 -- osd_op(mds.0.97:7 200.000081b0 [stat] 1.6f52aaeb rwordered e65927) v4 -- ?+0 0x1af3680 con 0x1ad69a0
  351. -103> 2013-09-17 16:39:26.076391 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.37 [2001:620:0:6::106]:6821/5118 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (3592419272 0 2312406677) 0x1a2f400 con 0x1ad6580
  352. -102> 2013-09-17 16:39:26.077044 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10a]:6807/15040
  353. -101> 2013-09-17 16:39:26.077058 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10a]:6816/16018
  354. -100> 2013-09-17 16:39:26.077400 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.61 [2001:620:0:6::10c]:6816/5936 1 ==== osd_op_reply(1 mds0_inotable [read 0~34] ondisk = 0) v4 ==== 112+0+34 (271308853 0 409257965) 0x1a2fc00 con 0x1a35760
  355. -99> 2013-09-17 16:39:26.089235 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.38 [2001:620:0:6::10a]:6807/15040 1 ==== osd_op_reply(6 200.000081af [stat] ondisk = 0) v4 ==== 111+0+16 (1519692392 0 1135597311) 0x1a2fe00 con 0x1ad6c60
  356. -98> 2013-09-17 16:39:26.096561 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.46 [2001:620:0:6::10a]:6816/16018 1 ==== osd_op_reply(7 200.000081b0 [stat] ack = -2 (No such file or directory)) v4 ==== 111+0+0 (2944366586 0 0) 0x1afc000 con 0x1ad69a0
  357. -97> 2013-09-17 16:39:26.096604 7f7937f63700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 139247905477 (header had 139247862597). recovered.
  358. -96> 2013-09-17 16:39:26.096613 7f7937f63700 2 mds.0.97 boot_start 2: loading/discovering base inodes
  359. -95> 2013-09-17 16:39:26.096641 7f7937f63700 0 mds.0.cache creating system inode with ino:100
  360. -94> 2013-09-17 16:39:26.096791 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6803/4506 -- osd_op(mds.0.97:8 100.00000000 [getxattr inode] 1.c5265ab3 e65927) v4 -- ?+0 0x1b03d40 con 0x1ad7ce0
  361. -93> 2013-09-17 16:39:26.096877 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10e]:6821/6014 -- osd_op(mds.0.97:9 100.00000000.inode [read 0~0] 1.85dde07f e65927) v4 -- ?+0 0x1b03b00 con 0x1ad7a20
  362. -92> 2013-09-17 16:39:26.096903 7f7937f63700 0 mds.0.cache creating system inode with ino:1
  363. -91> 2013-09-17 16:39:26.096961 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.97:10 1.00000000 [getxattr inode] 1.6b2cdaff e65927) v4 -- ?+0 0x1b03680 con 0x1ad7760
  364. -90> 2013-09-17 16:39:26.097013 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::102]:6818/5149 -- osd_op(mds.0.97:11 1.00000000.inode [read 0~0] 1.232c0e14 e65927) v4 -- ?+0 0x1b03440 con 0x1ad6840
  365. -89> 2013-09-17 16:39:26.097904 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::106]:6803/4506
  366. -88> 2013-09-17 16:39:26.097976 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10e]:6821/6014
  367. -87> 2013-09-17 16:39:26.098141 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.26 [2001:620:0:6::102]:6818/5149 2 ==== osd_op_reply(11 1.00000000.inode [read 0~432] ondisk = 0) v4 ==== 115+0+432 (3547013857 0 4176654171) 0x1a2fc00 con 0x1ad6840
  368. -86> 2013-09-17 16:39:26.098408 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::110]:6803/4416
  369. -85> 2013-09-17 16:39:26.099656 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.17 [2001:620:0:6::110]:6803/4416 1 ==== osd_op_reply(10 1.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 109+0+0 (4124585041 0 0) 0x1a2fe00 con 0x1ad7760
  370. -84> 2013-09-17 16:39:26.122736 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.54 [2001:620:0:6::10e]:6821/6014 1 ==== osd_op_reply(9 100.00000000.inode [read 0~432] ondisk = 0) v4 ==== 117+0+432 (4100013422 0 1444729992) 0x1b64000 con 0x1ad7a20
  371. -83> 2013-09-17 16:39:26.126999 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.30 [2001:620:0:6::106]:6803/4506 1 ==== osd_op_reply(8 100.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 111+0+0 (3079986607 0 0) 0x1b64200 con 0x1ad7ce0
  372. -82> 2013-09-17 16:39:26.127045 7f7937f63700 2 mds.0.97 boot_start 3: replaying mds log
  373. -81> 2013-09-17 16:39:26.127274 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.97:12 200.00008191 [read 2612~4191692] 1.239e4e7a e65927) v4 -- ?+0 0x1b626c0 con 0x1b28f20
  374. -80> 2013-09-17 16:39:26.127318 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.97:13 200.00008192 [read 0~4194304] 1.a4dfc87a e65927) v4 -- ?+0 0x1b62b40 con 0x1ad7760
  375. -79> 2013-09-17 16:39:26.127378 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::112]:6809/5287 -- osd_op(mds.0.97:14 200.00008193 [read 0~4194304] 1.2b2e6e44 e65927) v4 -- ?+0 0x1b62900 con 0x1b29080
  376. -78> 2013-09-17 16:39:26.127405 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.97:15 200.00008194 [read 0~4194304] 1.21189b05 e65927) v4 -- ?+0 0x1b63200 con 0x1b28f20
  377. -77> 2013-09-17 16:39:26.127452 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10e]:6824/6100 -- osd_op(mds.0.97:16 200.00008195 [read 0~4194304] 1.15998a83 e65927) v4 -- ?+0 0x1b62fc0 con 0x1b294a0
  378. -76> 2013-09-17 16:39:26.127504 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.97:17 200.00008196 [read 0~4194304] 1.6e722afe e65927) v4 -- ?+0 0x1b62d80 con 0x1b29b80
  379. -75> 2013-09-17 16:39:26.127547 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10c]:6822/6179 -- osd_op(mds.0.97:18 200.00008197 [read 0~4194304] 1.afc91a02 e65927) v4 -- ?+0 0x1b63440 con 0x1b298c0
  380. -74> 2013-09-17 16:39:26.127595 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6806/4599 -- osd_op(mds.0.97:19 200.00008198 [read 0~4194304] 1.57b56ff9 e65927) v4 -- ?+0 0x1b62b40 con 0x1b2a420
  381. -73> 2013-09-17 16:39:26.127618 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10c]:6822/6179 -- osd_op(mds.0.97:20 200.00008199 [read 0~4194304] 1.e1451abb e65927) v4 -- ?+0 0x1b03680 con 0x1b298c0
  382. -72> 2013-09-17 16:39:26.127655 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6800/4401 -- osd_op(mds.0.97:21 200.0000819a [read 0~4194304] 1.3b3e5ba2 e65927) v4 -- ?+0 0x1b03440 con 0x1b2a160
  383. -71> 2013-09-17 16:39:26.127707 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::108]:6800/4326 -- osd_op(mds.0.97:22 200.0000819b [read 0~4194304] 1.6fd1d5b1 e65927) v4 -- ?+0 0x1b63d40 con 0x1b29ce0
  384. -70> 2013-09-17 16:39:26.128433 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::106]:6812/4773
  385. -69> 2013-09-17 16:39:26.128500 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10e]:6824/6100
  386. -68> 2013-09-17 16:39:26.128556 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::106]:6806/4599
  387. -67> 2013-09-17 16:39:26.128572 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::106]:6800/4401
  388. -66> 2013-09-17 16:39:26.128899 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::100]:6803/17838
  389. -65> 2013-09-17 16:39:26.128917 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10c]:6822/6179
  390. -64> 2013-09-17 16:39:26.128931 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::112]:6809/5287
  391. -63> 2013-09-17 16:39:26.129485 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::108]:6800/4326
  392. -62> 2013-09-17 16:39:26.150520 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.31 [2001:620:0:6::106]:6806/4599 1 ==== osd_op_reply(19 200.00008198 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (602267768 0 3553492181) 0x1b64400 con 0x1b2a420
  393. -61> 2013-09-17 16:39:26.152297 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.17 [2001:620:0:6::110]:6803/4416 2 ==== osd_op_reply(13 200.00008192 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3063332071 0 2526270159) 0x1b64800 con 0x1ad7760
  394. -60> 2013-09-17 16:39:26.159180 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.55 [2001:620:0:6::10e]:6824/6100 1 ==== osd_op_reply(16 200.00008195 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1241099650 0 2609303889) 0x1a2fc00 con 0x1b294a0
  395. -59> 2013-09-17 16:39:26.160926 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.29 [2001:620:0:6::106]:6800/4401 1 ==== osd_op_reply(21 200.0000819a [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (834410933 0 2142534846) 0x1a2fe00 con 0x1b2a160
  396. -58> 2013-09-17 16:39:26.163112 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.63 [2001:620:0:6::10c]:6822/6179 1 ==== osd_op_reply(18 200.00008197 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3294847089 0 1276237556) 0x1b64000 con 0x1b298c0
  397. -57> 2013-09-17 16:39:26.165508 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.59 [2001:620:0:6::112]:6809/5287 1 ==== osd_op_reply(14 200.00008193 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1655049703 0 3935915300) 0x1b64200 con 0x1b29080
  398. -56> 2013-09-17 16:39:26.175062 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.63 [2001:620:0:6::10c]:6822/6179 2 ==== osd_op_reply(20 200.00008199 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4578763 0 3932121001) 0x1b64800 con 0x1b298c0
  399. -55> 2013-09-17 16:39:26.175113 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.33 [2001:620:0:6::106]:6812/4773 1 ==== osd_op_reply(17 200.00008196 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (61348016 0 2711816313) 0x1b64a00 con 0x1b29b80
  400. -54> 2013-09-17 16:39:26.176897 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.10 [2001:620:0:6::108]:6800/4326 1 ==== osd_op_reply(22 200.0000819b [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (900561672 0 2378898454) 0x1b64c00 con 0x1b29ce0
  401. -53> 2013-09-17 16:39:26.180306 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.1 [2001:620:0:6::100]:6803/17838 1 ==== osd_op_reply(15 200.00008194 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3463786648 0 878672217) 0x1a2fc00 con 0x1b28f20
  402. -52> 2013-09-17 16:39:26.195572 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.1 [2001:620:0:6::100]:6803/17838 2 ==== osd_op_reply(12 200.00008191 [read 2612~4191692] ondisk = 0) v4 ==== 111+0+4191692 (867354642 0 2662385302) 0x1b64000 con 0x1b28f20
  403. -51> 2013-09-17 16:39:26.221249 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10c]:6825/6343 -- osd_op(mds.0.97:23 200.0000819c [read 0~4194304] 1.77a62f6b e65927) v4 -- ?+0 0x1b638c0 con 0x1b2ab00
  404. -50> 2013-09-17 16:39:26.249005 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10c]:6825/6343
  405. -49> 2013-09-17 16:39:26.249226 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::112]:6809/5287 -- osd_op(mds.0.97:24 200.0000819d [read 0~4194304] 1.107e78d e65927) v4 -- ?+0 0x1b85d40 con 0x1b29080
  406. -48> 2013-09-17 16:39:26.250912 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.64 [2001:620:0:6::10c]:6825/6343 1 ==== osd_op_reply(23 200.0000819c [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3781841810 0 2150641809) 0x1b64800 con 0x1b2ab00
  407. -47> 2013-09-17 16:39:26.276878 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6821/5118 -- osd_op(mds.0.97:25 200.0000819e [read 0~4194304] 1.a2930100 e65927) v4 -- ?+0 0x1b858c0 con 0x1ad6580
  408. -46> 2013-09-17 16:39:26.305098 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10e]:6821/6014 -- osd_op(mds.0.97:26 200.0000819f [read 0~4194304] 1.ed1e40de e65927) v4 -- ?+0 0x1ff4b40 con 0x1ad7a20
  409. -45> 2013-09-17 16:39:26.313050 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.59 [2001:620:0:6::112]:6809/5287 2 ==== osd_op_reply(24 200.0000819d [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1422488744 0 2295832098) 0x1b64c00 con 0x1b29080
  410. -44> 2013-09-17 16:39:26.333147 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::104]:6806/25076 -- osd_op(mds.0.97:27 200.000081a0 [read 0~4194304] 1.dda38fb1 e65927) v4 -- ?+0 0x1ff46c0 con 0x1b2a840
  411. -43> 2013-09-17 16:39:26.341033 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::104]:6806/25076
  412. -42> 2013-09-17 16:39:26.341057 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.37 [2001:620:0:6::106]:6821/5118 2 ==== osd_op_reply(25 200.0000819e [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3607174410 0 3634468612) 0x1b65000 con 0x1ad6580
  413. -41> 2013-09-17 16:39:26.349104 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.54 [2001:620:0:6::10e]:6821/6014 2 ==== osd_op_reply(26 200.0000819f [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1915570235 0 3358981880) 0x1b65400 con 0x1ad7a20
  414. -40> 2013-09-17 16:39:26.361318 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::104]:6806/25076 -- osd_op(mds.0.97:28 200.000081a1 [read 0~4194304] 1.f800871d e65927) v4 -- ?+0 0x1ff4240 con 0x1b2a840
  415. -39> 2013-09-17 16:39:26.389101 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6816/16018 -- osd_op(mds.0.97:29 200.000081a2 [read 0~4194304] 1.a4a00eea e65927) v4 -- ?+0 0x1b84b40 con 0x1ad69a0
  416. -38> 2013-09-17 16:39:26.417110 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::102]:6803/4235 -- osd_op(mds.0.97:30 200.000081a3 [read 0~4194304] 1.4f5b5f19 e65927) v4 -- ?+0 0x1ff4b40 con 0x1b2a580
  417. -37> 2013-09-17 16:39:26.445003 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::102]:6803/4235
  418. -36> 2013-09-17 16:39:26.445056 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.5 [2001:620:0:6::104]:6806/25076 1 ==== osd_op_reply(27 200.000081a0 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2587444315 0 1515010739) 0x1b65600 con 0x1b2a840
  419. -35> 2013-09-17 16:39:26.445142 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6822/16408 -- osd_op(mds.0.97:31 200.000081a4 [read 0~4194304] 1.fcd3b7a4 e65927) v4 -- ?+0 0x1ff46c0 con 0x1b2bce0
  420. -34> 2013-09-17 16:39:26.472773 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.97:32 200.000081a5 [read 0~4194304] 1.83e11fc2 e65927) v4 -- ?+0 0x1a20d80 con 0x1b29b80
  421. -33> 2013-09-17 16:39:26.500505 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6822/16408 -- osd_op(mds.0.97:33 200.000081a6 [read 0~4194304] 1.51bbed43 e65927) v4 -- ?+0 0x1b03b00 con 0x1b2bce0
  422. -32> 2013-09-17 16:39:26.525045 7f7935e5e700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  423. -31> 2013-09-17 16:39:26.525059 7f7935e5e700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6789/0 -- mdsbeacon(182994/ineri up:replay seq 4 v711) v2 -- ?+0 0x1a78840 con 0x1a34f20
  424. -30> 2013-09-17 16:39:26.525076 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10a]:6822/16408
  425. -29> 2013-09-17 16:39:26.525098 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.21 [2001:620:0:6::102]:6803/4235 1 ==== osd_op_reply(30 200.000081a3 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (347040060 0 1663964) 0x2c3e000 con 0x1b2a580
  426. -28> 2013-09-17 16:39:26.528692 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.97:34 200.000081a7 [read 0~4194304] 1.6dd25d51 e65927) v4 -- ?+0 0x1ff5b00 con 0x1ad7760
  427. -27> 2013-09-17 16:39:26.533028 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== mon.0 [2001:620:0:6::106]:6789/0 19 ==== mdsbeacon(182994/ineri up:replay seq 4 v711) v2 ==== 107+0+0 (2768802075 0 0) 0x1a78dc0 con 0x1a34f20
  428. -26> 2013-09-17 16:39:26.556689 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::112]:6803/5058 -- osd_op(mds.0.97:35 200.000081a8 [read 0~4194304] 1.ef569196 e65927) v4 -- ?+0 0x1ff5680 con 0x2016c60
  429. -25> 2013-09-17 16:39:26.585203 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.97:36 200.000081a9 [read 0~4194304] 1.9a15953a e65927) v4 -- ?+0 0x1ff5200 con 0x1b28f20
  430. -24> 2013-09-17 16:39:26.613467 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10e]:6824/6100 -- osd_op(mds.0.97:37 200.000081aa [read 0~4194304] 1.6c6009b4 e65927) v4 -- ?+0 0x1b62d80 con 0x1b294a0
  431. -23> 2013-09-17 16:39:26.641648 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::112]:6803/5058
  432. -22> 2013-09-17 16:39:26.641667 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.42 [2001:620:0:6::10a]:6822/16408 1 ==== osd_op_reply(31 200.000081a4 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1978700092 0 950685490) 0x2c3e200 con 0x1b2bce0
  433. -21> 2013-09-17 16:39:26.641698 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.57 [2001:620:0:6::112]:6803/5058 1 ==== osd_op_reply(35 200.000081a8 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3938402198 0 2756327013) 0x1a2fc00 con 0x2016c60
  434. -20> 2013-09-17 16:39:26.641710 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.46 [2001:620:0:6::10a]:6816/16018 2 ==== osd_op_reply(29 200.000081a2 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3753480872 0 2531503823) 0x1b65e00 con 0x1ad69a0
  435. -19> 2013-09-17 16:39:26.641725 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.17 [2001:620:0:6::110]:6803/4416 3 ==== osd_op_reply(34 200.000081a7 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1893917975 0 640228953) 0x1b64600 con 0x1ad7760
  436. -18> 2013-09-17 16:39:26.641739 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.1 [2001:620:0:6::100]:6803/17838 3 ==== osd_op_reply(36 200.000081a9 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (774835425 0 1428356768) 0x1a2fe00 con 0x1b28f20
  437. -17> 2013-09-17 16:39:26.641751 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.55 [2001:620:0:6::10e]:6824/6100 2 ==== osd_op_reply(37 200.000081aa [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3243917460 0 1683739580) 0x1b64800 con 0x1b294a0
  438. -16> 2013-09-17 16:39:26.641765 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.33 [2001:620:0:6::106]:6812/4773 2 ==== osd_op_reply(32 200.000081a5 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (250572758 0 1239038857) 0x2c3e600 con 0x1b29b80
  439. -15> 2013-09-17 16:39:26.641779 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.5 [2001:620:0:6::104]:6806/25076 2 ==== osd_op_reply(28 200.000081a1 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (326566679 0 4024501100) 0x1b65a00 con 0x1b2a840
  440. -14> 2013-09-17 16:39:26.641794 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.42 [2001:620:0:6::10a]:6822/16408 2 ==== osd_op_reply(33 200.000081a6 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1975340032 0 3695802034) 0x2c3ea00 con 0x1b2bce0
  441. -13> 2013-09-17 16:39:26.641866 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6819/15996 -- osd_op(mds.0.97:38 200.000081ab [read 0~4194304] 1.2bfafcbd e65927) v4 -- ?+0 0x2c5dd40 con 0x20169a0
  442. -12> 2013-09-17 16:39:26.669904 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::108]:6809/8955 -- osd_op(mds.0.97:39 200.000081ac [read 0~4194304] 1.2b1c95db e65927) v4 -- ?+0 0x2c5d8c0 con 0x20166e0
  443. -11> 2013-09-17 16:39:26.697700 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::108]:6812/10542 -- osd_op(mds.0.97:40 200.000081ad [read 0~4194304] 1.4dc097e4 e65927) v4 -- ?+0 0x2c5d440 con 0x2016420
  444. -10> 2013-09-17 16:39:26.725604 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.97:41 200.000081ae [read 0~4194304] 1.115a19cf e65927) v4 -- ?+0 0x2c5cfc0 con 0x1b29b80
  445. -9> 2013-09-17 16:39:26.741007 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::10a]:6819/15996
  446. -8> 2013-09-17 16:39:26.753372 7f7934249700 1 -- [2001:620:0:6::d0]:6800/15023 --> [2001:620:0:6::10a]:6807/15040 -- osd_op(mds.0.97:42 200.000081af [read 0~1206981] 1.e01c7684 e65927) v4 -- ?+0 0x1af26c0 con 0x1ad6c60
  447. -7> 2013-09-17 16:39:26.821004 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::108]:6809/8955
  448. -6> 2013-09-17 16:39:26.861004 7f7937f63700 5 mds.0.97 ms_handle_connect on [2001:620:0:6::108]:6812/10542
  449. -5> 2013-09-17 16:39:26.861017 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.40 [2001:620:0:6::10a]:6819/15996 1 ==== osd_op_reply(38 200.000081ab [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1807843691 0 3103833645) 0x1a2fc00 con 0x20169a0
  450. -4> 2013-09-17 16:39:26.909038 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.8 [2001:620:0:6::108]:6809/8955 1 ==== osd_op_reply(39 200.000081ac [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1669560530 0 277755) 0x1a2fe00 con 0x20166e0
  451. -3> 2013-09-17 16:39:26.949244 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.9 [2001:620:0:6::108]:6812/10542 1 ==== osd_op_reply(40 200.000081ad [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1054656727 0 626561037) 0x1b65e00 con 0x2016420
  452. -2> 2013-09-17 16:39:26.977317 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.38 [2001:620:0:6::10a]:6807/15040 2 ==== osd_op_reply(42 200.000081af [read 0~1206981] ondisk = 0) v4 ==== 111+0+1206981 (2470569257 0 119534278) 0x1b64800 con 0x1ad6c60
  453. -1> 2013-09-17 16:39:27.001018 7f7937f63700 1 -- [2001:620:0:6::d0]:6800/15023 <== osd.33 [2001:620:0:6::106]:6812/4773 3 ==== osd_op_reply(41 200.000081ae [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3247869483 0 3604356958) 0x2c3e400 con 0x1b29b80
  454. 0> 2013-09-17 16:39:27.036674 7f7934249700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f7934249700 time 2013-09-17 16:39:27.035936
  455. mds/journal.cc: 1170: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  456.  
  457. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  458. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  459. 2: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  460. 3: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  461. 4: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  462. 5: (()+0x7e9a) [0x7f793cb57e9a]
  463. 6: (clone()+0x6d) [0x7f793b9f4cbd]
  464. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  465.  
  466. --- logging levels ---
  467. 0/ 5 none
  468. 0/ 1 lockdep
  469. 0/ 1 context
  470. 1/ 1 crush
  471. 1/ 5 mds
  472. 1/ 5 mds_balancer
  473. 1/ 5 mds_locker
  474. 1/ 5 mds_log
  475. 1/ 5 mds_log_expire
  476. 1/ 5 mds_migrator
  477. 0/ 1 buffer
  478. 0/ 1 timer
  479. 0/ 1 filer
  480. 0/ 1 striper
  481. 0/ 1 objecter
  482. 0/ 5 rados
  483. 0/ 5 rbd
  484. 0/ 5 journaler
  485. 0/ 5 objectcacher
  486. 0/ 5 client
  487. 0/ 5 osd
  488. 0/ 5 optracker
  489. 0/ 5 objclass
  490. 1/ 3 filestore
  491. 1/ 3 journal
  492. 0/ 5 ms
  493. 1/ 5 mon
  494. 0/10 monc
  495. 0/ 5 paxos
  496. 0/ 5 tp
  497. 1/ 5 auth
  498. 1/ 5 crypto
  499. 1/ 1 finisher
  500. 1/ 5 heartbeatmap
  501. 1/ 5 perfcounter
  502. 1/ 5 rgw
  503. 1/ 5 hadoop
  504. 1/ 5 javaclient
  505. 1/ 5 asok
  506. 1/ 1 throttle
  507. -2/-2 (syslog threshold)
  508. -1/-1 (stderr threshold)
  509. max_recent 10000
  510. max_new 1000
  511. log_file /var/log/ceph/ceph-mds.ineri.log
  512. --- end dump of recent events ---
  513. 2013-09-17 16:39:27.039529 7f7934249700 -1 *** Caught signal (Aborted) **
  514. in thread 7f7934249700
  515.  
  516. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  517. 1: /usr/bin/ceph-mds() [0x864f00]
  518. 2: (()+0xfcb0) [0x7f793cb5fcb0]
  519. 3: (gsignal()+0x35) [0x7f793b937425]
  520. 4: (abort()+0x17b) [0x7f793b93ab8b]
  521. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f793c232e2d]
  522. 6: (()+0x5ef26) [0x7f793c230f26]
  523. 7: (()+0x5ef53) [0x7f793c230f53]
  524. 8: (()+0x5f17e) [0x7f793c23117e]
  525. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cb11d]
  526. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  527. 11: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  528. 12: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  529. 13: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  530. 14: (()+0x7e9a) [0x7f793cb57e9a]
  531. 15: (clone()+0x6d) [0x7f793b9f4cbd]
  532. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  533.  
  534. --- begin dump of recent events ---
  535. 0> 2013-09-17 16:39:27.039529 7f7934249700 -1 *** Caught signal (Aborted) **
  536. in thread 7f7934249700
  537.  
  538. ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  539. 1: /usr/bin/ceph-mds() [0x864f00]
  540. 2: (()+0xfcb0) [0x7f793cb5fcb0]
  541. 3: (gsignal()+0x35) [0x7f793b937425]
  542. 4: (abort()+0x17b) [0x7f793b93ab8b]
  543. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f793c232e2d]
  544. 6: (()+0x5ef26) [0x7f793c230f26]
  545. 7: (()+0x5ef53) [0x7f793c230f53]
  546. 8: (()+0x5f17e) [0x7f793c23117e]
  547. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7cb11d]
  548. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x522ee5]
  549. 11: (EUpdate::replay(MDS*)+0x3a) [0x52ba3a]
  550. 12: (MDLog::_replay_thread()+0x5cf) [0x6f392f]
  551. 13: (MDLog::ReplayThread::entry()+0xd) [0x505a9d]
  552. 14: (()+0x7e9a) [0x7f793cb57e9a]
  553. 15: (clone()+0x6d) [0x7f793b9f4cbd]
  554. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  555.  
  556. --- logging levels ---
  557. 0/ 5 none
  558. 0/ 1 lockdep
  559. 0/ 1 context
  560. 1/ 1 crush
  561. 1/ 5 mds
  562. 1/ 5 mds_balancer
  563. 1/ 5 mds_locker
  564. 1/ 5 mds_log
  565. 1/ 5 mds_log_expire
  566. 1/ 5 mds_migrator
  567. 0/ 1 buffer
  568. 0/ 1 timer
  569. 0/ 1 filer
  570. 0/ 1 striper
  571. 0/ 1 objecter
  572. 0/ 5 rados
  573. 0/ 5 rbd
  574. 0/ 5 journaler
  575. 0/ 5 objectcacher
  576. 0/ 5 client
  577. 0/ 5 osd
  578. 0/ 5 optracker
  579. 0/ 5 objclass
  580. 1/ 3 filestore
  581. 1/ 3 journal
  582. 0/ 5 ms
  583. 1/ 5 mon
  584. 0/10 monc
  585. 0/ 5 paxos
  586. 0/ 5 tp
  587. 1/ 5 auth
  588. 1/ 5 crypto
  589. 1/ 1 finisher
  590. 1/ 5 heartbeatmap
  591. 1/ 5 perfcounter
  592. 1/ 5 rgw
  593. 1/ 5 hadoop
  594. 1/ 5 javaclient
  595. 1/ 5 asok
  596. 1/ 1 throttle
  597. -2/-2 (syslog threshold)
  598. -1/-1 (stderr threshold)
  599. max_recent 10000
  600. max_new 1000
  601. log_file /var/log/ceph/ceph-mds.ineri.log
  602. --- end dump of recent events ---
  603. 2013-09-17 16:41:42.114223 7f311709a780 0 ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b), process ceph-mds, pid 16608
  604. 2013-09-17 16:41:42.232406 7f3111b42700 1 mds.-1.0 handle_mds_map standby
  605. 2013-09-17 16:41:42.276282 7f3111b42700 1 mds.0.98 handle_mds_map i am now mds.0.98
  606. 2013-09-17 16:41:42.276285 7f3111b42700 1 mds.0.98 handle_mds_map state change up:standby --> up:replay
  607. 2013-09-17 16:41:42.276293 7f3111b42700 1 mds.0.98 replay_start
  608. 2013-09-17 16:41:42.276296 7f3111b42700 1 mds.0.98 recovery set is
  609. 2013-09-17 16:41:42.276297 7f3111b42700 1 mds.0.98 need osdmap epoch 65946, have 65945
  610. 2013-09-17 16:41:42.276298 7f3111b42700 1 mds.0.98 waiting for osdmap 65946 (which blacklists prior instance)
  611. 2013-09-17 16:41:42.301630 7f3111b42700 0 mds.0.cache creating system inode with ino:100
  612. 2013-09-17 16:41:42.301851 7f3111b42700 0 mds.0.cache creating system inode with ino:1
  613. 2013-09-17 16:41:43.502245 7f310de28700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f310de28700 time 2013-09-17 16:41:43.501501
  614. mds/journal.cc: 1173: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  615.  
  616. ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
  617. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x524105]
  618. 2: (EUpdate::replay(MDS*)+0x3a) [0x52cc3a]
  619. 3: (MDLog::_replay_thread()+0x5cf) [0x6f4d9f]
  620. 4: (MDLog::ReplayThread::entry()+0xd) [0x506cad]
  621. 5: (()+0x7e9a) [0x7f3116736e9a]
  622. 6: (clone()+0x6d) [0x7f31155d3cbd]
  623. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  624.  
  625. --- begin dump of recent events ---
  626. -226> 2013-09-17 16:41:42.112590 7f311709a780 5 asok(0x2416000) register_command perfcounters_dump hook 0x2402010
  627. -225> 2013-09-17 16:41:42.112615 7f311709a780 5 asok(0x2416000) register_command 1 hook 0x2402010
  628. -224> 2013-09-17 16:41:42.112616 7f311709a780 5 asok(0x2416000) register_command perf dump hook 0x2402010
  629. -223> 2013-09-17 16:41:42.112621 7f311709a780 5 asok(0x2416000) register_command perfcounters_schema hook 0x2402010
  630. -222> 2013-09-17 16:41:42.112623 7f311709a780 5 asok(0x2416000) register_command 2 hook 0x2402010
  631. -221> 2013-09-17 16:41:42.112624 7f311709a780 5 asok(0x2416000) register_command perf schema hook 0x2402010
  632. -220> 2013-09-17 16:41:42.112626 7f311709a780 5 asok(0x2416000) register_command config show hook 0x2402010
  633. -219> 2013-09-17 16:41:42.112628 7f311709a780 5 asok(0x2416000) register_command config set hook 0x2402010
  634. -218> 2013-09-17 16:41:42.112629 7f311709a780 5 asok(0x2416000) register_command log flush hook 0x2402010
  635. -217> 2013-09-17 16:41:42.112631 7f311709a780 5 asok(0x2416000) register_command log dump hook 0x2402010
  636. -216> 2013-09-17 16:41:42.112632 7f311709a780 5 asok(0x2416000) register_command log reopen hook 0x2402010
  637. -215> 2013-09-17 16:41:42.114223 7f311709a780 0 ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b), process ceph-mds, pid 16608
  638. -214> 2013-09-17 16:41:42.114712 7f311709a780 1 accepter.accepter.bind my_inst.addr is [::]:6800/16608 need_addr=1
  639. -213> 2013-09-17 16:41:42.115393 7f311709a780 1 finished global_init_daemonize
  640. -212> 2013-09-17 16:41:42.155424 7f311709a780 5 asok(0x2416000) init /var/run/ceph/ceph-mds.ineri.asok
  641. -211> 2013-09-17 16:41:42.155481 7f311709a780 5 asok(0x2416000) bind_and_listen /var/run/ceph/ceph-mds.ineri.asok
  642. -210> 2013-09-17 16:41:42.155521 7f311709a780 5 asok(0x2416000) register_command 0 hook 0x24000b8
  643. -209> 2013-09-17 16:41:42.155561 7f311709a780 5 asok(0x2416000) register_command version hook 0x24000b8
  644. -208> 2013-09-17 16:41:42.155568 7f311709a780 5 asok(0x2416000) register_command git_version hook 0x24000b8
  645. -207> 2013-09-17 16:41:42.155574 7f311709a780 5 asok(0x2416000) register_command help hook 0x24020d0
  646. -206> 2013-09-17 16:41:42.155633 7f311709a780 10 monclient(hunting): build_initial_monmap
  647. -205> 2013-09-17 16:41:42.155648 7f3113345700 5 asok(0x2416000) entry start
  648. -204> 2013-09-17 16:41:42.155752 7f311709a780 1 -- [::]:6800/16608 messenger.start
  649. -203> 2013-09-17 16:41:42.155889 7f311709a780 5 adding auth protocol: cephx
  650. -202> 2013-09-17 16:41:42.155903 7f311709a780 5 adding auth protocol: cephx
  651. -201> 2013-09-17 16:41:42.156025 7f311709a780 1 accepter.accepter.start
  652. -200> 2013-09-17 16:41:42.156085 7f311709a780 10 monclient(hunting): init
  653. -199> 2013-09-17 16:41:42.156098 7f311709a780 5 adding auth protocol: cephx
  654. -198> 2013-09-17 16:41:42.156100 7f311709a780 10 monclient(hunting): auth_supported 2 method cephx
  655. -197> 2013-09-17 16:41:42.164908 7f311709a780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ineri/keyring
  656. -196> 2013-09-17 16:41:42.165016 7f311709a780 10 monclient(hunting): _reopen_session
  657. -195> 2013-09-17 16:41:42.165117 7f311709a780 10 monclient(hunting): _pick_new_mon picked mon.h1 con 0x2428f20 addr [2001:620:0:6::106]:6789/0
  658. -194> 2013-09-17 16:41:42.165179 7f311709a780 10 monclient(hunting): _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  659. -193> 2013-09-17 16:41:42.165186 7f311709a780 1 -- [::]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x2414480 con 0x2428f20
  660. -192> 2013-09-17 16:41:42.165205 7f311709a780 10 monclient(hunting): renew_subs
  661. -191> 2013-09-17 16:41:42.165589 7f3117098700 1 -- [2001:620:0:6::d0]:6800/16608 learned my addr [2001:620:0:6::d0]:6800/16608
  662. -190> 2013-09-17 16:41:42.165868 7f3111b42700 5 mds.-1.0 ms_handle_connect on [2001:620:0:6::106]:6789/0
  663. -189> 2013-09-17 16:41:42.166321 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 1 ==== mon_map v1 ==== 476+0+0 (335251837 0 0) 0x24883c0 con 0x2428f20
  664. -188> 2013-09-17 16:41:42.166348 7f3111b42700 10 monclient(hunting): handle_monmap mon_map v1
  665. -187> 2013-09-17 16:41:42.166360 7f3111b42700 10 monclient(hunting): got monmap 10, mon.h1 is now rank 0
  666. -186> 2013-09-17 16:41:42.166363 7f3111b42700 10 monclient(hunting): dump:
  667. epoch 10
  668. fsid 2fe1a358-7f88-4a1b-a31b-ba7501870c80
  669. last_changed 2013-09-09 10:59:40.635175
  670. created 2013-01-28 06:53:12.936035
  671. 0: [2001:620:0:6::106]:6789/0 mon.h1
  672. 1: [2001:620:0:6::10c]:6789/0 mon.h5
  673. 2: [2001:620:0:6::10e]:6789/0 mon.h3
  674.  
  675. -185> 2013-09-17 16:41:42.166405 7f3111b42700 1 monclient(hunting): found mon.h1
  676. -184> 2013-09-17 16:41:42.166415 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2901454495 0 0) 0x2423400 con 0x2428f20
  677. -183> 2013-09-17 16:41:42.166442 7f3111b42700 10 monclient: my global_id is 183000
  678. -182> 2013-09-17 16:41:42.166573 7f3111b42700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  679. -181> 2013-09-17 16:41:42.166581 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x24146c0 con 0x2428f20
  680. -180> 2013-09-17 16:41:42.167122 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3329010122 0 0) 0x2423800 con 0x2428f20
  681. -179> 2013-09-17 16:41:42.167195 7f3111b42700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  682. -178> 2013-09-17 16:41:42.167203 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2414b40 con 0x2428f20
  683. -177> 2013-09-17 16:41:42.167893 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (1304400589 0 0) 0x2423600 con 0x2428f20
  684. -176> 2013-09-17 16:41:42.167962 7f3111b42700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  685. -175> 2013-09-17 16:41:42.167969 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x245e380 con 0x2428f20
  686. -174> 2013-09-17 16:41:42.167990 7f3111b42700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-09-17 16:41:12.167990)
  687. -173> 2013-09-17 16:41:42.167995 7f3111b42700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  688. -172> 2013-09-17 16:41:42.167998 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2414900 con 0x2428f20
  689. -171> 2013-09-17 16:41:42.168030 7f311709a780 5 monclient: authenticate success, global_id 183000
  690. -170> 2013-09-17 16:41:42.168044 7f311709a780 10 monclient: wait_auth_rotating waiting (until 2013-09-17 16:42:12.168042)
  691. -169> 2013-09-17 16:41:42.168325 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 5 ==== mon_map v1 ==== 476+0+0 (335251837 0 0) 0x24885a0 con 0x2428f20
  692. -168> 2013-09-17 16:41:42.168340 7f3111b42700 10 monclient: handle_monmap mon_map v1
  693. -167> 2013-09-17 16:41:42.168347 7f3111b42700 10 monclient: got monmap 10, mon.h1 is now rank 0
  694. -166> 2013-09-17 16:41:42.168349 7f3111b42700 10 monclient: dump:
  695. epoch 10
  696. fsid 2fe1a358-7f88-4a1b-a31b-ba7501870c80
  697. last_changed 2013-09-09 10:59:40.635175
  698. created 2013-01-28 06:53:12.936035
  699. 0: [2001:620:0:6::106]:6789/0 mon.h1
  700. 1: [2001:620:0:6::10c]:6789/0 mon.h5
  701. 2: [2001:620:0:6::10e]:6789/0 mon.h3
  702.  
  703. -165> 2013-09-17 16:41:42.168370 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x245e540 con 0x2428f20
  704. -164> 2013-09-17 16:41:42.168383 7f3111b42700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:41:42.165214 renew after 2013-09-17 16:44:12.165214
  705. -163> 2013-09-17 16:41:42.168399 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1811832681 0 0) 0x2423a00 con 0x2428f20
  706. -162> 2013-09-17 16:41:42.168434 7f3111b42700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-09-17 16:41:12.168434)
  707. -161> 2013-09-17 16:41:42.168481 7f311709a780 10 monclient: wait_auth_rotating done
  708. -160> 2013-09-17 16:41:42.168515 7f311709a780 5 asok(0x2416000) register_command objecter_requests hook 0x2402100
  709. -159> 2013-09-17 16:41:42.168574 7f311709a780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  710. -158> 2013-09-17 16:41:42.168587 7f311709a780 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- mdsbeacon(183000/ineri up:boot seq 1 v0) v2 -- ?+0 0x246c2c0 con 0x2428f20
  711. -157> 2013-09-17 16:41:42.168631 7f311709a780 10 monclient: renew_subs
  712. -156> 2013-09-17 16:41:42.168636 7f311709a780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  713. -155> 2013-09-17 16:41:42.168641 7f311709a780 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({monmap=11+,osdmap=0}) v2 -- ?+0 0x245e1c0 con 0x2428f20
  714. -154> 2013-09-17 16:41:42.168658 7f311709a780 10 monclient: renew_subs
  715. -153> 2013-09-17 16:41:42.168662 7f311709a780 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  716. -152> 2013-09-17 16:41:42.168668 7f311709a780 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({mdsmap=0+,monmap=11+,osdmap=0}) v2 -- ?+0 0x245ea80 con 0x2428f20
  717. -151> 2013-09-17 16:41:42.176338 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 8 ==== osd_map(65945..65945 src has 24657..65945) v3 ==== 42253+0+0 (3966277932 0 0) 0x2414fc0 con 0x2428f20
  718. -150> 2013-09-17 16:41:42.176634 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x245e1c0 con 0x2428f20
  719. -149> 2013-09-17 16:41:42.176647 7f3111b42700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:41:42.168635 renew after 2013-09-17 16:44:12.168635
  720. -148> 2013-09-17 16:41:42.176656 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 10 ==== mdsmap(e 718) v1 ==== 595+0+0 (1593107590 0 0) 0x2423800 con 0x2428f20
  721. -147> 2013-09-17 16:41:42.176663 7f3111b42700 5 mds.-1.0 handle_mds_map epoch 718 from mon.0
  722. -146> 2013-09-17 16:41:42.176723 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 11 ==== osd_map(65945..65945 src has 24657..65945) v3 ==== 42253+0+0 (3966277932 0 0) 0x2415440 con 0x2428f20
  723. -145> 2013-09-17 16:41:42.176739 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x245e380 con 0x2428f20
  724. -144> 2013-09-17 16:41:42.176745 7f3111b42700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  725. -143> 2013-09-17 16:41:42.232328 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 13 ==== mdsmap(e 719) v1 ==== 587+0+0 (3142798676 0 0) 0x2423400 con 0x2428f20
  726. -142> 2013-09-17 16:41:42.232371 7f3111b42700 5 mds.-1.-1 handle_mds_map epoch 719 from mon.0
  727. -141> 2013-09-17 16:41:42.232400 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 mark_down [2001:620:0:6::d0]:6800/15023 -- pipe dne
  728. -140> 2013-09-17 16:41:42.232406 7f3111b42700 1 mds.-1.0 handle_mds_map standby
  729. -139> 2013-09-17 16:41:42.276211 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 14 ==== mdsmap(e 720) v1 ==== 595+0+0 (2280804946 0 0) 0x2423c00 con 0x2428f20
  730. -138> 2013-09-17 16:41:42.276256 7f3111b42700 5 mds.-1.0 handle_mds_map epoch 720 from mon.0
  731. -137> 2013-09-17 16:41:42.276282 7f3111b42700 1 mds.0.98 handle_mds_map i am now mds.0.98
  732. -136> 2013-09-17 16:41:42.276285 7f3111b42700 1 mds.0.98 handle_mds_map state change up:standby --> up:replay
  733. -135> 2013-09-17 16:41:42.276293 7f3111b42700 1 mds.0.98 replay_start
  734. -134> 2013-09-17 16:41:42.276296 7f3111b42700 1 mds.0.98 recovery set is
  735. -133> 2013-09-17 16:41:42.276297 7f3111b42700 1 mds.0.98 need osdmap epoch 65946, have 65945
  736. -132> 2013-09-17 16:41:42.276298 7f3111b42700 1 mds.0.98 waiting for osdmap 65946 (which blacklists prior instance)
  737. -131> 2013-09-17 16:41:42.276305 7f3111b42700 10 monclient: renew_subs
  738. -130> 2013-09-17 16:41:42.276309 7f3111b42700 10 monclient: _send_mon_message to mon.h1 at [2001:620:0:6::106]:6789/0
  739. -129> 2013-09-17 16:41:42.276313 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6789/0 -- mon_subscribe({mdsmap=721+,monmap=11+,osdmap=65946}) v2 -- ?+0 0x245e380 con 0x2428f20
  740. -128> 2013-09-17 16:41:42.276893 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 15 ==== osd_map(65946..65946 src has 24657..65946) v3 ==== 308+0+0 (25826674 0 0) 0x2415200 con 0x2428f20
  741. -127> 2013-09-17 16:41:42.276942 7f3111b42700 2 mds.0.98 boot_start 1: opening inotable
  742. -126> 2013-09-17 16:41:42.277083 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10c]:6816/5936 -- osd_op(mds.0.98:1 mds0_inotable [read 0~0] 1.b852b893 e65946) v4 -- ?+0 0x2415440 con 0x2429760
  743. -125> 2013-09-17 16:41:42.277102 7f3111b42700 2 mds.0.98 boot_start 1: opening sessionmap
  744. -124> 2013-09-17 16:41:42.277157 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::102]:6809/4481 -- osd_op(mds.0.98:2 mds0_sessionmap [read 0~0] 1.3270c60b e65946) v4 -- ?+0 0x2414fc0 con 0x24c8160
  745. -123> 2013-09-17 16:41:42.277168 7f3111b42700 2 mds.0.98 boot_start 1: opening anchor table
  746. -122> 2013-09-17 16:41:42.277201 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::102]:6818/5149 -- osd_op(mds.0.98:3 mds_anchortable [read 0~0] 1.a977f6a7 e65946) v4 -- ?+0 0x2414d80 con 0x24c8840
  747. -121> 2013-09-17 16:41:42.277212 7f3111b42700 2 mds.0.98 boot_start 1: opening snap table
  748. -120> 2013-09-17 16:41:42.277254 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6821/5118 -- osd_op(mds.0.98:4 mds_snaptable [read 0~0] 1.d90270ad e65946) v4 -- ?+0 0x24146c0 con 0x24c8580
  749. -119> 2013-09-17 16:41:42.277269 7f3111b42700 2 mds.0.98 boot_start 1: opening mds log
  750. -118> 2013-09-17 16:41:42.277272 7f3111b42700 5 mds.0.log open discovering log bounds
  751. -117> 2013-09-17 16:41:42.277276 7f3111b42700 1 mds.0.journaler(ro) recover start
  752. -116> 2013-09-17 16:41:42.277277 7f3111b42700 1 mds.0.journaler(ro) read_head
  753. -115> 2013-09-17 16:41:42.277315 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10e]:6800/5420 -- osd_op(mds.0.98:5 200.00000000 [read 0~0] 1.844f3494 e65946) v4 -- ?+0 0x2414480 con 0x24c8f20
  754. -114> 2013-09-17 16:41:42.277334 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== mon.0 [2001:620:0:6::106]:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1089595926 0 0) 0x245ec40 con 0x2428f20
  755. -113> 2013-09-17 16:41:42.277341 7f3111b42700 10 monclient: handle_subscribe_ack sent 2013-09-17 16:41:42.276307 renew after 2013-09-17 16:44:12.276307
  756. -112> 2013-09-17 16:41:42.278327 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10c]:6816/5936
  757. -111> 2013-09-17 16:41:42.278522 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::106]:6821/5118
  758. -110> 2013-09-17 16:41:42.278557 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::102]:6818/5149
  759. -109> 2013-09-17 16:41:42.278611 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10e]:6800/5420
  760. -108> 2013-09-17 16:41:42.278626 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::102]:6809/4481
  761. -107> 2013-09-17 16:41:42.279748 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.23 [2001:620:0:6::102]:6809/4481 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~1938] ondisk = 0) v4 ==== 114+0+1938 (4116860547 0 1042821830) 0x2423e00 con 0x24c8160
  762. -106> 2013-09-17 16:41:42.287726 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.47 [2001:620:0:6::10e]:6800/5420 1 ==== osd_op_reply(5 200.00000000 [read 0~84] ondisk = 0) v4 ==== 111+0+84 (999552900 0 1876858853) 0x24ee000 con 0x24c8f20
  763. -105> 2013-09-17 16:41:42.287778 7f3111b42700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 139120869376, expire 139120871988, write 139247862597). probing for end of log (from 139247862597)...
  764. -104> 2013-09-17 16:41:42.287785 7f3111b42700 1 mds.0.journaler(ro) probing for end of the log
  765. -103> 2013-09-17 16:41:42.287885 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6807/15040 -- osd_op(mds.0.98:6 200.000081af [stat] 1.e01c7684 rwordered e65946) v4 -- ?+0 0x24e98c0 con 0x24c8c60
  766. -102> 2013-09-17 16:41:42.287950 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6816/16018 -- osd_op(mds.0.98:7 200.000081b0 [stat] 1.6f52aaeb rwordered e65946) v4 -- ?+0 0x24e9680 con 0x24c89a0
  767. -101> 2013-09-17 16:41:42.287969 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.26 [2001:620:0:6::102]:6818/5149 1 ==== osd_op_reply(3 mds_anchortable [read 0~2058] ondisk = 0) v4 ==== 114+0+2058 (3577373924 0 1136844878) 0x24ee200 con 0x24c8840
  768. -100> 2013-09-17 16:41:42.289154 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10a]:6816/16018
  769. -99> 2013-09-17 16:41:42.289168 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10a]:6807/15040
  770. -98> 2013-09-17 16:41:42.289183 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.37 [2001:620:0:6::106]:6821/5118 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (250758173 0 2312406677) 0x2423400 con 0x24c8580
  771. -97> 2013-09-17 16:41:42.292429 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.61 [2001:620:0:6::10c]:6816/5936 1 ==== osd_op_reply(1 mds0_inotable [read 0~34] ondisk = 0) v4 ==== 112+0+34 (3368446944 0 409257965) 0x2423c00 con 0x2429760
  772. -96> 2013-09-17 16:41:42.297595 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.38 [2001:620:0:6::10a]:6807/15040 1 ==== osd_op_reply(6 200.000081af [stat] ondisk = 0) v4 ==== 111+0+16 (2188974525 0 1135597311) 0x2423e00 con 0x24c8c60
  773. -95> 2013-09-17 16:41:42.301578 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.46 [2001:620:0:6::10a]:6816/16018 1 ==== osd_op_reply(7 200.000081b0 [stat] ack = -2 (No such file or directory)) v4 ==== 111+0+0 (2006107183 0 0) 0x24ee000 con 0x24c89a0
  774. -94> 2013-09-17 16:41:42.301619 7f3111b42700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 139247905477 (header had 139247862597). recovered.
  775. -93> 2013-09-17 16:41:42.301625 7f3111b42700 2 mds.0.98 boot_start 2: loading/discovering base inodes
  776. -92> 2013-09-17 16:41:42.301630 7f3111b42700 0 mds.0.cache creating system inode with ino:100
  777. -91> 2013-09-17 16:41:42.301769 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6803/4506 -- osd_op(mds.0.98:8 100.00000000 [getxattr inode] 1.c5265ab3 e65946) v4 -- ?+0 0x24e9200 con 0x24c9ce0
  778. -90> 2013-09-17 16:41:42.301838 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10e]:6821/6014 -- osd_op(mds.0.98:9 100.00000000.inode [read 0~0] 1.85dde07f e65946) v4 -- ?+0 0x24f5d40 con 0x24c9a20
  779. -89> 2013-09-17 16:41:42.301851 7f3111b42700 0 mds.0.cache creating system inode with ino:1
  780. -88> 2013-09-17 16:41:42.301896 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.98:10 1.00000000 [getxattr inode] 1.6b2cdaff e65946) v4 -- ?+0 0x24f58c0 con 0x24c9760
  781. -87> 2013-09-17 16:41:42.301920 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::102]:6818/5149 -- osd_op(mds.0.98:11 1.00000000.inode [read 0~0] 1.232c0e14 e65946) v4 -- ?+0 0x24f5680 con 0x24c8840
  782. -86> 2013-09-17 16:41:42.302902 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.26 [2001:620:0:6::102]:6818/5149 2 ==== osd_op_reply(11 1.00000000.inode [read 0~432] ondisk = 0) v4 ==== 115+0+432 (193368372 0 4176654171) 0x2423c00 con 0x24c8840
  783. -85> 2013-09-17 16:41:42.302930 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::106]:6803/4506
  784. -84> 2013-09-17 16:41:42.303115 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10e]:6821/6014
  785. -83> 2013-09-17 16:41:42.303455 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::110]:6803/4416
  786. -82> 2013-09-17 16:41:42.308955 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.54 [2001:620:0:6::10e]:6821/6014 1 ==== osd_op_reply(9 100.00000000.inode [read 0~432] ondisk = 0) v4 ==== 117+0+432 (747409083 0 1444729992) 0x2423e00 con 0x24c9a20
  787. -81> 2013-09-17 16:41:42.312372 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.30 [2001:620:0:6::106]:6803/4506 1 ==== osd_op_reply(8 100.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 111+0+0 (1870229114 0 0) 0x2554000 con 0x24c9ce0
  788. -80> 2013-09-17 16:41:42.609789 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.17 [2001:620:0:6::110]:6803/4416 1 ==== osd_op_reply(10 1.00000000 [getxattr] ondisk = -61 (No data available)) v4 ==== 109+0+0 (758488964 0 0) 0x2554200 con 0x24c9760
  789. -79> 2013-09-17 16:41:42.609865 7f3111b42700 2 mds.0.98 boot_start 3: replaying mds log
  790. -78> 2013-09-17 16:41:42.610136 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.98:12 200.00008191 [read 2612~4191692] 1.239e4e7a e65946) v4 -- ?+0 0x25526c0 con 0x2515080
  791. -77> 2013-09-17 16:41:42.610197 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.98:13 200.00008192 [read 0~4194304] 1.a4dfc87a e65946) v4 -- ?+0 0x2552b40 con 0x24c9760
  792. -76> 2013-09-17 16:41:42.610273 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::112]:6809/5287 -- osd_op(mds.0.98:14 200.00008193 [read 0~4194304] 1.2b2e6e44 e65946) v4 -- ?+0 0x2552900 con 0x25151e0
  793. -75> 2013-09-17 16:41:42.610316 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.98:15 200.00008194 [read 0~4194304] 1.21189b05 e65946) v4 -- ?+0 0x2553200 con 0x2515080
  794. -74> 2013-09-17 16:41:42.610427 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10e]:6824/6100 -- osd_op(mds.0.98:16 200.00008195 [read 0~4194304] 1.15998a83 e65946) v4 -- ?+0 0x2552fc0 con 0x2515600
  795. -73> 2013-09-17 16:41:42.610564 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.98:17 200.00008196 [read 0~4194304] 1.6e722afe e65946) v4 -- ?+0 0x2552d80 con 0x2515ce0
  796. -72> 2013-09-17 16:41:42.610728 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10c]:6822/6179 -- osd_op(mds.0.98:18 200.00008197 [read 0~4194304] 1.afc91a02 e65946) v4 -- ?+0 0x2553440 con 0x2515a20
  797. -71> 2013-09-17 16:41:42.610834 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6806/4599 -- osd_op(mds.0.98:19 200.00008198 [read 0~4194304] 1.57b56ff9 e65946) v4 -- ?+0 0x2552b40 con 0x2515e40
  798. -70> 2013-09-17 16:41:42.610874 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10c]:6822/6179 -- osd_op(mds.0.98:20 200.00008199 [read 0~4194304] 1.e1451abb e65946) v4 -- ?+0 0x24f58c0 con 0x2515a20
  799. -69> 2013-09-17 16:41:42.611007 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6800/4401 -- osd_op(mds.0.98:21 200.0000819a [read 0~4194304] 1.3b3e5ba2 e65946) v4 -- ?+0 0x24f5680 con 0x2564580
  800. -68> 2013-09-17 16:41:42.611164 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::108]:6800/4326 -- osd_op(mds.0.98:22 200.0000819b [read 0~4194304] 1.6fd1d5b1 e65946) v4 -- ?+0 0x2553d40 con 0x25642c0
  801. -67> 2013-09-17 16:41:42.611452 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10e]:6824/6100
  802. -66> 2013-09-17 16:41:42.611465 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::112]:6809/5287
  803. -65> 2013-09-17 16:41:42.611743 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::106]:6812/4773
  804. -64> 2013-09-17 16:41:42.611815 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10c]:6822/6179
  805. -63> 2013-09-17 16:41:42.612009 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::100]:6803/17838
  806. -62> 2013-09-17 16:41:42.612144 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::106]:6806/4599
  807. -61> 2013-09-17 16:41:42.612157 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::106]:6800/4401
  808. -60> 2013-09-17 16:41:42.612353 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::108]:6800/4326
  809. -59> 2013-09-17 16:41:42.636637 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.63 [2001:620:0:6::10c]:6822/6179 1 ==== osd_op_reply(20 200.00008199 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3634916894 0 3932121001) 0x2554400 con 0x2515a20
  810. -58> 2013-09-17 16:41:42.638675 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.29 [2001:620:0:6::106]:6800/4401 1 ==== osd_op_reply(21 200.0000819a [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3914443360 0 2142534846) 0x2554600 con 0x2564580
  811. -57> 2013-09-17 16:41:42.639940 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.59 [2001:620:0:6::112]:6809/5287 1 ==== osd_op_reply(14 200.00008193 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3125526066 0 3935915300) 0x2423c00 con 0x25151e0
  812. -56> 2013-09-17 16:41:42.640195 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.55 [2001:620:0:6::10e]:6824/6100 1 ==== osd_op_reply(16 200.00008195 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2434014807 0 2609303889) 0x2423e00 con 0x2515600
  813. -55> 2013-09-17 16:41:42.641602 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.31 [2001:620:0:6::106]:6806/4599 1 ==== osd_op_reply(19 200.00008198 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4211626925 0 3553492181) 0x31b0000 con 0x2515e40
  814. -54> 2013-09-17 16:41:42.641647 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.10 [2001:620:0:6::108]:6800/4326 1 ==== osd_op_reply(22 200.0000819b [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3980445917 0 2378898454) 0x31b0200 con 0x25642c0
  815. -53> 2013-09-17 16:41:42.642427 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.33 [2001:620:0:6::106]:6812/4773 1 ==== osd_op_reply(17 200.00008196 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3678776165 0 2711816313) 0x31b0400 con 0x2515ce0
  816. -52> 2013-09-17 16:41:42.645584 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.1 [2001:620:0:6::100]:6803/17838 1 ==== osd_op_reply(15 200.00008194 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (379093837 0 878672217) 0x31b0600 con 0x2515080
  817. -51> 2013-09-17 16:41:42.650261 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.63 [2001:620:0:6::10c]:6822/6179 2 ==== osd_op_reply(18 200.00008197 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (479122340 0 1276237556) 0x31b0a00 con 0x2515a20
  818. -50> 2013-09-17 16:41:42.659609 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.1 [2001:620:0:6::100]:6803/17838 2 ==== osd_op_reply(12 200.00008191 [read 2612~4191692] ondisk = 0) v4 ==== 111+0+4191692 (3948903367 0 2662385302) 0x2423e00 con 0x2515080
  819. -49> 2013-09-17 16:41:42.668323 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.17 [2001:620:0:6::110]:6803/4416 2 ==== osd_op_reply(13 200.00008192 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1853558578 0 2526270159) 0x31b0400 con 0x24c9760
  820. -48> 2013-09-17 16:41:42.692190 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10c]:6825/6343 -- osd_op(mds.0.98:23 200.0000819c [read 0~4194304] 1.77a62f6b e65946) v4 -- ?+0 0x2579d40 con 0x2564000
  821. -47> 2013-09-17 16:41:42.697005 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10c]:6825/6343
  822. -46> 2013-09-17 16:41:42.711032 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.64 [2001:620:0:6::10c]:6825/6343 1 ==== osd_op_reply(23 200.0000819c [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (965206087 0 2150641809) 0x31b0600 con 0x2564000
  823. -45> 2013-09-17 16:41:42.717319 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::112]:6809/5287 -- osd_op(mds.0.98:24 200.0000819d [read 0~4194304] 1.107e78d e65946) v4 -- ?+0 0x25798c0 con 0x25151e0
  824. -44> 2013-09-17 16:41:42.745033 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.59 [2001:620:0:6::112]:6809/5287 2 ==== osd_op_reply(24 200.0000819d [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2351220605 0 2295832098) 0x31b0c00 con 0x25151e0
  825. -43> 2013-09-17 16:41:42.747905 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6821/5118 -- osd_op(mds.0.98:25 200.0000819e [read 0~4194304] 1.a2930100 e65946) v4 -- ?+0 0x2579440 con 0x24c8580
  826. -42> 2013-09-17 16:41:42.774880 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10e]:6821/6014 -- osd_op(mds.0.98:26 200.0000819f [read 0~4194304] 1.ed1e40de e65946) v4 -- ?+0 0x31e0b40 con 0x24c9a20
  827. -41> 2013-09-17 16:41:42.801902 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::104]:6806/25076 -- osd_op(mds.0.98:27 200.000081a0 [read 0~4194304] 1.dda38fb1 e65946) v4 -- ?+0 0x31e06c0 con 0x2565a20
  828. -40> 2013-09-17 16:41:42.829197 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::104]:6806/25076 -- osd_op(mds.0.98:28 200.000081a1 [read 0~4194304] 1.f800871d e65946) v4 -- ?+0 0x25798c0 con 0x2565a20
  829. -39> 2013-09-17 16:41:42.856316 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6816/16018 -- osd_op(mds.0.98:29 200.000081a2 [read 0~4194304] 1.a4a00eea e65946) v4 -- ?+0 0x25538c0 con 0x24c89a0
  830. -38> 2013-09-17 16:41:42.869027 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::104]:6806/25076
  831. -37> 2013-09-17 16:41:42.869040 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.37 [2001:620:0:6::106]:6821/5118 2 ==== osd_op_reply(25 200.0000819e [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (267169503 0 3634468612) 0x31b1000 con 0x24c8580
  832. -36> 2013-09-17 16:41:42.883596 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::102]:6803/4235 -- osd_op(mds.0.98:30 200.000081a3 [read 0~4194304] 1.4f5b5f19 e65946) v4 -- ?+0 0x31e0b40 con 0x2565760
  833. -35> 2013-09-17 16:41:42.915692 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6822/16408 -- osd_op(mds.0.98:31 200.000081a4 [read 0~4194304] 1.fcd3b7a4 e65946) v4 -- ?+0 0x2579440 con 0x25654a0
  834. -34> 2013-09-17 16:41:42.942678 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.98:32 200.000081a5 [read 0~4194304] 1.83e11fc2 e65946) v4 -- ?+0 0x24146c0 con 0x2515ce0
  835. -33> 2013-09-17 16:41:42.969873 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6822/16408 -- osd_op(mds.0.98:33 200.000081a6 [read 0~4194304] 1.51bbed43 e65946) v4 -- ?+0 0x2414d80 con 0x25654a0
  836. -32> 2013-09-17 16:41:42.981004 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::102]:6803/4235
  837. -31> 2013-09-17 16:41:42.988176 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10a]:6822/16408
  838. -30> 2013-09-17 16:41:42.988194 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.54 [2001:620:0:6::10e]:6821/6014 2 ==== osd_op_reply(26 200.0000819f [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2864774126 0 3358981880) 0x31b1400 con 0x24c9a20
  839. -29> 2013-09-17 16:41:42.997066 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::110]:6803/4416 -- osd_op(mds.0.98:34 200.000081a7 [read 0~4194304] 1.6dd25d51 e65946) v4 -- ?+0 0x31e1b00 con 0x24c9760
  840. -28> 2013-09-17 16:41:43.028132 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::112]:6803/5058 -- osd_op(mds.0.98:35 200.000081a8 [read 0~4194304] 1.ef569196 e65946) v4 -- ?+0 0x31e1680 con 0x29b6c60
  841. -27> 2013-09-17 16:41:43.055023 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::100]:6803/17838 -- osd_op(mds.0.98:36 200.000081a9 [read 0~4194304] 1.9a15953a e65946) v4 -- ?+0 0x31e1200 con 0x2515080
  842. -26> 2013-09-17 16:41:43.082313 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::112]:6803/5058
  843. -25> 2013-09-17 16:41:43.082332 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.33 [2001:620:0:6::106]:6812/4773 2 ==== osd_op_reply(32 200.000081a5 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3590511619 0 1239038857) 0x29b2600 con 0x2515ce0
  844. -24> 2013-09-17 16:41:43.082354 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.5 [2001:620:0:6::104]:6806/25076 1 ==== osd_op_reply(27 200.000081a0 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (1121253262 0 1515010739) 0x31b1600 con 0x2565a20
  845. -23> 2013-09-17 16:41:43.082369 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.21 [2001:620:0:6::102]:6803/4235 1 ==== osd_op_reply(30 200.000081a3 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3426933481 0 1663964) 0x29b2000 con 0x2565760
  846. -22> 2013-09-17 16:41:43.082392 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10e]:6824/6100 -- osd_op(mds.0.98:37 200.000081aa [read 0~4194304] 1.6c6009b4 e65946) v4 -- ?+0 0x24146c0 con 0x2515600
  847. -21> 2013-09-17 16:41:43.088840 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.42 [2001:620:0:6::10a]:6822/16408 1 ==== osd_op_reply(31 200.000081a4 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2904367849 0 950685490) 0x29b2200 con 0x25654a0
  848. -20> 2013-09-17 16:41:43.088932 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.57 [2001:620:0:6::112]:6803/5058 1 ==== osd_op_reply(35 200.000081a8 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (844270659 0 2756327013) 0x29b2c00 con 0x29b6c60
  849. -19> 2013-09-17 16:41:43.089015 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.46 [2001:620:0:6::10a]:6816/16018 2 ==== osd_op_reply(29 200.000081a2 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (122954109 0 2531503823) 0x31b1e00 con 0x24c89a0
  850. -18> 2013-09-17 16:41:43.089135 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.17 [2001:620:0:6::110]:6803/4416 3 ==== osd_op_reply(34 200.000081a7 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2819577538 0 640228953) 0x31b0200 con 0x24c9760
  851. -17> 2013-09-17 16:41:43.090386 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.1 [2001:620:0:6::100]:6803/17838 3 ==== osd_op_reply(36 200.000081a9 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (4139958068 0 1428356768) 0x2423c00 con 0x2515080
  852. -16> 2013-09-17 16:41:43.090525 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.5 [2001:620:0:6::104]:6806/25076 2 ==== osd_op_reply(28 200.000081a1 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3415912642 0 4024501100) 0x31b1a00 con 0x2565a20
  853. -15> 2013-09-17 16:41:43.090611 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.42 [2001:620:0:6::10a]:6822/16408 2 ==== osd_op_reply(33 200.000081a6 [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (2907766741 0 3695802034) 0x29b2a00 con 0x25654a0
  854. -14> 2013-09-17 16:41:43.095793 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.55 [2001:620:0:6::10e]:6824/6100 2 ==== osd_op_reply(37 200.000081aa [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (431485761 0 1683739580) 0x31b0400 con 0x2515600
  855. -13> 2013-09-17 16:41:43.112005 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6819/15996 -- osd_op(mds.0.98:38 200.000081ab [read 0~4194304] 1.2bfafcbd e65946) v4 -- ?+0 0x2dd1d40 con 0x29b69a0
  856. -12> 2013-09-17 16:41:43.139106 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::108]:6809/8955 -- osd_op(mds.0.98:39 200.000081ac [read 0~4194304] 1.2b1c95db e65946) v4 -- ?+0 0x2dd18c0 con 0x29b66e0
  857. -11> 2013-09-17 16:41:43.166145 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::108]:6812/10542 -- osd_op(mds.0.98:40 200.000081ad [read 0~4194304] 1.4dc097e4 e65946) v4 -- ?+0 0x2dd1440 con 0x29b6420
  858. -10> 2013-09-17 16:41:43.181008 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::10a]:6819/15996
  859. -9> 2013-09-17 16:41:43.181022 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::108]:6809/8955
  860. -8> 2013-09-17 16:41:43.185003 7f3111b42700 5 mds.0.98 ms_handle_connect on [2001:620:0:6::108]:6812/10542
  861. -7> 2013-09-17 16:41:43.185013 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.40 [2001:620:0:6::10a]:6819/15996 1 ==== osd_op_reply(38 200.000081ab [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3006018238 0 3103833645) 0x31b0600 con 0x29b69a0
  862. -6> 2013-09-17 16:41:43.193054 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.8 [2001:620:0:6::108]:6809/8955 1 ==== osd_op_reply(39 200.000081ac [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3144600327 0 277755) 0x31b0c00 con 0x29b66e0
  863. -5> 2013-09-17 16:41:43.198489 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::106]:6812/4773 -- osd_op(mds.0.98:41 200.000081ae [read 0~4194304] 1.115a19cf e65946) v4 -- ?+0 0x2dd0fc0 con 0x2515ce0
  864. -4> 2013-09-17 16:41:43.225701 7f310de28700 1 -- [2001:620:0:6::d0]:6800/16608 --> [2001:620:0:6::10a]:6807/15040 -- osd_op(mds.0.98:42 200.000081af [read 0~1206981] 1.e01c7684 e65946) v4 -- ?+0 0x24e8480 con 0x24c8c60
  865. -3> 2013-09-17 16:41:43.389049 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.9 [2001:620:0:6::108]:6812/10542 1 ==== osd_op_reply(40 200.000081ad [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (3862002434 0 626561037) 0x2423c00 con 0x29b6420
  866. -2> 2013-09-17 16:41:43.441057 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.38 [2001:620:0:6::10a]:6807/15040 2 ==== osd_op_reply(42 200.000081af [read 0~1206981] ondisk = 0) v4 ==== 111+0+1206981 (1269585660 0 119534278) 0x31b0200 con 0x24c8c60
  867. -1> 2013-09-17 16:41:43.471552 7f3111b42700 1 -- [2001:620:0:6::d0]:6800/16608 <== osd.33 [2001:620:0:6::106]:6812/4773 3 ==== osd_op_reply(41 200.000081ae [read 0~4194304] ondisk = 0) v4 ==== 111+0+4194304 (427507198 0 3604356958) 0x29b2400 con 0x2515ce0
  868. 0> 2013-09-17 16:41:43.502245 7f310de28700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f310de28700 time 2013-09-17 16:41:43.501501
  869. mds/journal.cc: 1173: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))
  870.  
  871. ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
  872. 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x524105]
  873. 2: (EUpdate::replay(MDS*)+0x3a) [0x52cc3a]
  874. 3: (MDLog::_replay_thread()+0x5cf) [0x6f4d9f]
  875. 4: (MDLog::ReplayThread::entry()+0xd) [0x506cad]
  876. 5: (()+0x7e9a) [0x7f3116736e9a]
  877. 6: (clone()+0x6d) [0x7f31155d3cbd]
  878. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  879.  
  880. --- logging levels ---
  881. 0/ 5 none
  882. 0/ 1 lockdep
  883. 0/ 1 context
  884. 1/ 1 crush
  885. 1/ 5 mds
  886. 1/ 5 mds_balancer
  887. 1/ 5 mds_locker
  888. 1/ 5 mds_log
  889. 1/ 5 mds_log_expire
  890. 1/ 5 mds_migrator
  891. 0/ 1 buffer
  892. 0/ 1 timer
  893. 0/ 1 filer
  894. 0/ 1 striper
  895. 0/ 1 objecter
  896. 0/ 5 rados
  897. 0/ 5 rbd
  898. 0/ 5 journaler
  899. 0/ 5 objectcacher
  900. 0/ 5 client
  901. 0/ 5 osd
  902. 0/ 5 optracker
  903. 0/ 5 objclass
  904. 1/ 3 filestore
  905. 1/ 3 journal
  906. 0/ 5 ms
  907. 1/ 5 mon
  908. 0/10 monc
  909. 0/ 5 paxos
  910. 0/ 5 tp
  911. 1/ 5 auth
  912. 1/ 5 crypto
  913. 1/ 1 finisher
  914. 1/ 5 heartbeatmap
  915. 1/ 5 perfcounter
  916. 1/ 5 rgw
  917. 1/ 5 hadoop
  918. 1/ 5 javaclient
  919. 1/ 5 asok
  920. 1/ 1 throttle
  921. -2/-2 (syslog threshold)
  922. -1/-1 (stderr threshold)
  923. max_recent 10000
  924. max_new 1000
  925. log_file /var/log/ceph/ceph-mds.ineri.log
  926. --- end dump of recent events ---
  927. 2013-09-17 16:41:43.504539 7f310de28700 -1 *** Caught signal (Aborted) **
  928. in thread 7f310de28700
  929.  
  930. ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
  931. 1: /usr/bin/ceph-mds() [0x868f80]
  932. 2: (()+0xfcb0) [0x7f311673ecb0]
  933. 3: (gsignal()+0x35) [0x7f3115516425]
  934. 4: (abort()+0x17b) [0x7f3115519b8b]
  935. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3115e11e2d]
  936. 6: (()+0x5ef26) [0x7f3115e0ff26]
  937. 7: (()+0x5ef53) [0x7f3115e0ff53]
  938. 8: (()+0x5f17e) [0x7f3115e1017e]
  939. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7ce07d]
  940. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x524105]
  941. 11: (EUpdate::replay(MDS*)+0x3a) [0x52cc3a]
  942. 12: (MDLog::_replay_thread()+0x5cf) [0x6f4d9f]
  943. 13: (MDLog::ReplayThread::entry()+0xd) [0x506cad]
  944. 14: (()+0x7e9a) [0x7f3116736e9a]
  945. 15: (clone()+0x6d) [0x7f31155d3cbd]
  946. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  947.  
  948. --- begin dump of recent events ---
  949. 0> 2013-09-17 16:41:43.504539 7f310de28700 -1 *** Caught signal (Aborted) **
  950. in thread 7f310de28700
  951.  
  952. ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
  953. 1: /usr/bin/ceph-mds() [0x868f80]
  954. 2: (()+0xfcb0) [0x7f311673ecb0]
  955. 3: (gsignal()+0x35) [0x7f3115516425]
  956. 4: (abort()+0x17b) [0x7f3115519b8b]
  957. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3115e11e2d]
  958. 6: (()+0x5ef26) [0x7f3115e0ff26]
  959. 7: (()+0x5ef53) [0x7f3115e0ff53]
  960. 8: (()+0x5f17e) [0x7f3115e1017e]
  961. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x7ce07d]
  962. 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x3ac5) [0x524105]
  963. 11: (EUpdate::replay(MDS*)+0x3a) [0x52cc3a]
  964. 12: (MDLog::_replay_thread()+0x5cf) [0x6f4d9f]
  965. 13: (MDLog::ReplayThread::entry()+0xd) [0x506cad]
  966. 14: (()+0x7e9a) [0x7f3116736e9a]
  967. 15: (clone()+0x6d) [0x7f31155d3cbd]
  968. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  969.  
  970. --- logging levels ---
  971. 0/ 5 none
  972. 0/ 1 lockdep
  973. 0/ 1 context
  974. 1/ 1 crush
  975. 1/ 5 mds
  976. 1/ 5 mds_balancer
  977. 1/ 5 mds_locker
  978. 1/ 5 mds_log
  979. 1/ 5 mds_log_expire
  980. 1/ 5 mds_migrator
  981. 0/ 1 buffer
  982. 0/ 1 timer
  983. 0/ 1 filer
  984. 0/ 1 striper
  985. 0/ 1 objecter
  986. 0/ 5 rados
  987. 0/ 5 rbd
  988. 0/ 5 journaler
  989. 0/ 5 objectcacher
  990. 0/ 5 client
  991. 0/ 5 osd
  992. 0/ 5 optracker
  993. 0/ 5 objclass
  994. 1/ 3 filestore
  995. 1/ 3 journal
  996. 0/ 5 ms
  997. 1/ 5 mon
  998. 0/10 monc
  999. 0/ 5 paxos
  1000. 0/ 5 tp
  1001. 1/ 5 auth
  1002. 1/ 5 crypto
  1003. 1/ 1 finisher
  1004. 1/ 5 heartbeatmap
  1005. 1/ 5 perfcounter
  1006. 1/ 5 rgw
  1007. 1/ 5 hadoop
  1008. 1/ 5 javaclient
  1009. 1/ 5 asok
  1010. 1/ 1 throttle
  1011. -2/-2 (syslog threshold)
  1012. -1/-1 (stderr threshold)
  1013. max_recent 10000
  1014. max_new 1000
  1015. log_file /var/log/ceph/ceph-mds.ineri.log
  1016. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!