API tools faq
paste
Advertisement
xyasha

ceph-mds.a.log

xyasha
Apr 9th, 2013
168
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 80.95 KB | None | 0 0
raw download clone embed print report
  1. 2013-04-09 02:22:16.044730 7f587f88a760 10 mds.-1.0 168 MDSCacheObject
  2. 2013-04-09 02:22:16.044809 7f587f88a760 10 mds.-1.0 2144 CInode
  3. 2013-04-09 02:22:16.044825 7f587f88a760 10 mds.-1.0 16 elist<>::item *7=112
  4. 2013-04-09 02:22:16.044838 7f587f88a760 10 mds.-1.0 360 inode_t
  5. 2013-04-09 02:22:16.045432 7f587f88a760 10 mds.-1.0 56 nest_info_t
  6. 2013-04-09 02:22:16.045450 7f587f88a760 10 mds.-1.0 32 frag_info_t
  7. 2013-04-09 02:22:16.045463 7f587f88a760 10 mds.-1.0 40 SimpleLock *5=200
  8. 2013-04-09 02:22:16.045479 7f587f88a760 10 mds.-1.0 48 ScatterLock *3=144
  9. 2013-04-09 02:22:16.045492 7f587f88a760 10 mds.-1.0 472 CDentry
  10. 2013-04-09 02:22:16.045507 7f587f88a760 10 mds.-1.0 16 elist<>::item
  11. 2013-04-09 02:22:16.045520 7f587f88a760 10 mds.-1.0 40 SimpleLock
  12. 2013-04-09 02:22:16.045533 7f587f88a760 10 mds.-1.0 1016 CDir
  13. 2013-04-09 02:22:16.045545 7f587f88a760 10 mds.-1.0 16 elist<>::item *2=32
  14. 2013-04-09 02:22:16.045558 7f587f88a760 10 mds.-1.0 192 fnode_t
  15. 2013-04-09 02:22:16.045571 7f587f88a760 10 mds.-1.0 56 nest_info_t *2
  16. 2013-04-09 02:22:16.045583 7f587f88a760 10 mds.-1.0 32 frag_info_t *2
  17. 2013-04-09 02:22:16.045596 7f587f88a760 10 mds.-1.0 168 Capability
  18. 2013-04-09 02:22:16.045609 7f587f88a760 10 mds.-1.0 32 xlist<>::item *2=64
  19. 2013-04-09 02:22:16.046417 7f587f886700 10 mds.-1.0 MDS::ms_get_authorizer type=mon
  20. 2013-04-09 02:22:16.046737 7f587b640700 0 mds.-1.0 ms_handle_connect on mon.mon.mon.mon:6789/0
  21. 2013-04-09 02:22:16.047937 7f587f88a760 10 mds.-1.0 beacon_send up:boot seq 1 (currently up:boot)
  22. 2013-04-09 02:22:16.048041 7f587f88a760 10 mds.-1.0 create_logger
  23. 2013-04-09 02:22:16.427405 7f587b640700 5 mds.-1.0 handle_mds_map epoch 791 from mon.0
  24. 2013-04-09 02:22:16.427490 7f587b640700 10 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  25. 2013-04-09 02:22:16.427516 7f587b640700 10 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  26. 2013-04-09 02:22:16.427530 7f587b640700 10 mds.-1.0 map says i am mds.mds.mds.mds:6800/11967 mds.-1.0 state up:standby
  27. 2013-04-09 02:22:16.427549 7f587b640700 1 mds.-1.0 handle_mds_map standby
  28. 2013-04-09 02:22:20.048147 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 2 (currently up:standby)
  29. 2013-04-09 02:22:20.049368 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 2 rtt 0.001157
  30. 2013-04-09 02:22:21.048153 7f587953b700 20 mds.-1.bal get_load no root, no load
  31. 2013-04-09 02:22:21.048442 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  32. 2013-04-09 02:22:24.048497 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 3 (currently up:standby)
  33. 2013-04-09 02:22:24.049420 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 3 rtt 0.000845
  34. 2013-04-09 02:22:26.048306 7f587953b700 20 mds.-1.bal get_load no root, no load
  35. 2013-04-09 02:22:26.048473 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  36. 2013-04-09 02:22:28.048863 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 4 (currently up:standby)
  37. 2013-04-09 02:22:28.926165 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 4 rtt 0.877204
  38. 2013-04-09 02:22:31.048447 7f587953b700 20 mds.-1.bal get_load no root, no load
  39. 2013-04-09 02:22:31.048771 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  40. 2013-04-09 02:22:32.049446 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 5 (currently up:standby)
  41. 2013-04-09 02:22:32.702952 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 5 rtt 0.653394
  42. 2013-04-09 02:22:36.048607 7f587953b700 20 mds.-1.bal get_load no root, no load
  43. 2013-04-09 02:22:36.048786 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  44. 2013-04-09 02:22:36.049708 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 6 (currently up:standby)
  45. 2013-04-09 02:22:40.050077 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 7 (currently up:standby)
  46. 2013-04-09 02:22:41.048768 7f587953b700 20 mds.-1.bal get_load no root, no load
  47. 2013-04-09 02:22:41.049024 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  48. 2013-04-09 02:22:41.874961 7f587b640700 5 mds.-1.0 handle_mds_map epoch 792 from mon.0
  49. 2013-04-09 02:22:41.874989 7f587b640700 10 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  50. 2013-04-09 02:22:41.874997 7f587b640700 10 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  51. 2013-04-09 02:22:41.875003 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:replay
  52. 2013-04-09 02:22:41.875012 7f587b640700 10 mds.0.35 peer mds gid 7597 removed from map
  53. 2013-04-09 02:22:41.875023 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  54. 2013-04-09 02:22:41.875026 7f587b640700 1 mds.0.35 handle_mds_map state change up:standby --> up:replay
  55. 2013-04-09 02:22:41.875038 7f587b640700 1 mds.0.35 replay_start
  56. 2013-04-09 02:22:41.875079 7f587b640700 7 mds.0.cache set_recovery_set
  57. 2013-04-09 02:22:41.875109 7f587b640700 1 mds.0.35 recovery set is
  58. 2013-04-09 02:22:41.875113 7f587b640700 1 mds.0.35 need osdmap epoch 218, have 212
  59. 2013-04-09 02:22:41.875117 7f587b640700 1 mds.0.35 waiting for osdmap 218 (which blacklists prior instance)
  60. 2013-04-09 02:22:41.875179 7f587b640700 7 mds.0.cache handle_mds_failure mds.0
  61. 2013-04-09 02:22:41.875187 7f587b640700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are
  62. 2013-04-09 02:22:41.875191 7f587b640700 10 mds.0.cache wants_resolve
  63. 2013-04-09 02:22:41.875193 7f587b640700 10 mds.0.cache got_resolve
  64. 2013-04-09 02:22:41.875195 7f587b640700 10 mds.0.cache rejoin_sent
  65. 2013-04-09 02:22:41.875198 7f587b640700 10 mds.0.cache rejoin_gather
  66. 2013-04-09 02:22:41.875200 7f587b640700 10 mds.0.cache rejoin_ack_gather
  67. 2013-04-09 02:22:41.875206 7f587b640700 5 mds.0.migrator handle_mds_failure_or_stop mds.0
  68. 2013-04-09 02:22:41.875242 7f587b640700 10 mds.0.cache show_subtrees - no subtrees
  69. 2013-04-09 02:22:41.876347 7f587b640700 10 mds.0.35 handle_mds_beacon up:standby seq 6 rtt 5.826551
  70. 2013-04-09 02:22:41.877544 7f587b640700 10 mds.0.35 handle_mds_beacon up:standby seq 7 rtt 1.827345
  71. 2013-04-09 02:22:41.878594 7f587b640700 2 mds.0.35 boot_start 1: opening inotable
  72. 2013-04-09 02:22:41.878614 7f587b640700 10 mds.0.inotable: load
  73. 2013-04-09 02:22:41.878898 7f587b640700 2 mds.0.35 boot_start 1: opening sessionmap
  74. 2013-04-09 02:22:41.878908 7f587b640700 10 mds.0.sessionmap load
  75. 2013-04-09 02:22:41.879265 7f587b640700 2 mds.0.35 boot_start 1: opening anchor table
  76. 2013-04-09 02:22:41.879273 7f587b640700 10 mds.0.anchortable: load
  77. 2013-04-09 02:22:41.879332 7f587b640700 2 mds.0.35 boot_start 1: opening snap table
  78. 2013-04-09 02:22:41.879339 7f587b640700 10 mds.0.snaptable: load
  79. 2013-04-09 02:22:41.879382 7f587b640700 2 mds.0.35 boot_start 1: opening mds log
  80. 2013-04-09 02:22:41.879398 7f587b640700 5 mds.0.log open discovering log bounds
  81. 2013-04-09 02:22:41.880582 7f5878d3a700 10 mds.0.35 MDS::ms_get_authorizer type=osd
  82. 2013-04-09 02:22:41.881261 7f5878c39700 10 mds.0.35 MDS::ms_get_authorizer type=osd
  83. 2013-04-09 02:22:41.881724 7f587b640700 0 mds.0.35 ms_handle_connect on osd0.ods0.osd0.osd0:6800/12654
  84. 2013-04-09 02:22:41.881982 7f587b640700 0 mds.0.35 ms_handle_connect on osd1.ods1.osd1.osd1:6800/13459
  85. 2013-04-09 02:22:41.953572 7f587b640700 10 mds.0.inotable: load_2 got 29 bytes
  86. 2013-04-09 02:22:41.953577 7f587b640700 10 mds.0.inotable: load_2 loaded v0
  87. 2013-04-09 02:22:42.229260 7f587b640700 10 mds.0.sessionmap dump
  88. 2013-04-09 02:22:42.229265 7f587b640700 10 mds.0.sessionmap _load_finish v 0, 0 sessions, 17 bytes
  89. 2013-04-09 02:22:42.229267 7f587b640700 10 mds.0.sessionmap dump
  90. 2013-04-09 02:22:42.654548 7f587b640700 10 mds.0.anchortable: load_2 got 29 bytes
  91. 2013-04-09 02:22:42.654550 7f587b640700 10 mds.0.anchortable: load_2 loaded v0
  92. 2013-04-09 02:22:43.170973 7f587b640700 10 mds.0.snaptable: load_2 got 41 bytes
  93. 2013-04-09 02:22:43.170975 7f587b640700 10 mds.0.snaptable: load_2 loaded v0
  94. 2013-04-09 02:22:43.176378 7f587b640700 2 mds.0.35 boot_start 2: loading/discovering base inodes
  95. 2013-04-09 02:22:43.176383 7f587b640700 0 mds.0.cache creating system inode with ino:100
  96. 2013-04-09 02:22:43.176432 7f587b640700 10 mds.0.cache.ino(100) fetch
  97. 2013-04-09 02:22:43.176512 7f587b640700 0 mds.0.cache creating system inode with ino:1
  98. 2013-04-09 02:22:43.176517 7f587b640700 10 mds.0.cache.ino(1) fetch
  99. 2013-04-09 02:22:43.503350 7f587b640700 10 mds.0.cache.ino(100) _fetched got 0 and 399
  100. 2013-04-09 02:22:43.503365 7f587b640700 10 mds.0.cache.ino(100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  101. 2013-04-09 02:22:43.503393 7f587b640700 20 mds.0.cache.ino(100) decode_snap_blob snaprealm(100 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f5874018d20)
  102. 2013-04-09 02:22:43.503410 7f587b640700 10 mds.0.cache.ino(100) _fetched [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  103. 2013-04-09 02:22:44.050454 7f587953b700 10 mds.0.35 beacon_send up:replay seq 8 (currently up:replay)
  104. 2013-04-09 02:22:44.577154 7f587b640700 10 mds.0.cache.ino(1) _fetched got 0 and 428
  105. 2013-04-09 02:22:44.577160 7f587b640700 10 mds.0.cache.ino(1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  106. 2013-04-09 02:22:44.577169 7f587b640700 20 mds.0.cache.ino(1) decode_snap_blob snaprealm(1 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f587401b280)
  107. 2013-04-09 02:22:44.577174 7f587b640700 10 mds.0.cache.ino(1) _fetched [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) 0x7f5874008960]
  108. 2013-04-09 02:22:44.577184 7f587b640700 2 mds.0.35 boot_start 3: replaying mds log
  109. 2013-04-09 02:22:44.577187 7f587b640700 10 mds.0.log replay start, from 4195106 to 4195507
  110. 2013-04-09 02:22:44.577270 7f5878936700 10 mds.0.log _replay_thread start
  111. 2013-04-09 02:22:45.526865 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 8 rtt 1.476360
  112. 2013-04-09 02:22:46.048991 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  113. 2013-04-09 02:22:48.050750 7f587953b700 10 mds.0.35 beacon_send up:replay seq 9 (currently up:replay)
  114. 2013-04-09 02:22:49.410729 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 9 rtt 1.359929
  115. 2013-04-09 02:22:51.049179 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  116. 2013-04-09 02:22:52.050963 7f587953b700 10 mds.0.35 beacon_send up:replay seq 10 (currently up:replay)
  117. 2013-04-09 02:22:53.301578 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 10 rtt 1.250575
  118. 2013-04-09 02:22:54.494526 7f5878936700 10 mds.0.log _replay 4195106~397 / 4195507 2013-04-08 01:59:07.943387: ESubtreeMap 1 subtrees , 0 ambiguous [metablob 1, 1 dirs]
  119. 2013-04-09 02:22:54.494598 7f5878936700 10 mds.0.journal ESubtreeMap.replay -- reconstructing (auth) subtree spanning tree
  120. 2013-04-09 02:22:54.494613 7f5878936700 10 mds.0.journal EMetaBlob.replay 1 dirlumps by unknown.0
  121. 2013-04-09 02:22:54.494619 7f5878936700 10 mds.0.journal EMetaBlob.replay dir 1
  122. 2013-04-09 02:22:54.494631 7f5878936700 10 mds.0.cache force_dir_fragment * on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) 0x7f5874008960]
  123. 2013-04-09 02:22:54.494664 7f5878936700 10 mds.0.cache force_dir_fragment no frags under *
  124. 2013-04-09 02:22:54.494732 7f5878936700 7 mds.0.cache adjust_subtree_auth -1,-2 -> -2,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 0x1339d40]
  125. 2013-04-09 02:22:54.494754 7f5878936700 10 mds.0.cache show_subtrees - no subtrees
  126. 2013-04-09 02:22:54.494774 7f5878936700 7 mds.0.cache current root is [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  127. 2013-04-09 02:22:54.494788 7f5878936700 10 mds.0.cache.dir(1) setting dir_auth=-2,-2 from -1,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  128. 2013-04-09 02:22:54.494799 7f5878936700 10 mds.0.cache.dir(1) new subtree root, adjusting auth_pins
  129. 2013-04-09 02:22:54.494807 7f5878936700 15 mds.0.cache show_subtrees
  130. 2013-04-09 02:22:54.494817 7f5878936700 10 mds.0.cache |__-2 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  131. 2013-04-09 02:22:54.494834 7f5878936700 10 mds.0.journal EMetaBlob.replay added dir [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  132. 2013-04-09 02:22:54.494844 7f5878936700 10 mds.0.journal EMetaBlob.replay updated dir [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  133. 2013-04-09 02:22:54.494868 7f5878936700 7 mds.0.cache adjust_bounded_subtree_auth -2,-2 -> 0,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] bound_dfs []
  134. 2013-04-09 02:22:54.494886 7f5878936700 7 mds.0.cache adjust_bounded_subtree_auth -2,-2 -> 0,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] bounds
  135. 2013-04-09 02:22:54.494904 7f5878936700 15 mds.0.cache show_subtrees
  136. 2013-04-09 02:22:54.494913 7f5878936700 10 mds.0.cache |__-2 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  137. 2013-04-09 02:22:54.494929 7f5878936700 7 mds.0.cache current root is [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  138. 2013-04-09 02:22:54.494936 7f5878936700 10 mds.0.cache.dir(1) setting dir_auth=0,-2 from -2,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  139. 2013-04-09 02:22:54.494948 7f5878936700 15 mds.0.cache show_subtrees
  140. 2013-04-09 02:22:54.494952 7f5878936700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  141. 2013-04-09 02:22:54.494960 7f5878936700 15 mds.0.cache show_subtrees
  142. 2013-04-09 02:22:54.494963 7f5878936700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  143. 2013-04-09 02:22:54.494978 7f5878936700 10 mds.0.log _replay - complete, 1 events
  144. 2013-04-09 02:22:54.494983 7f5878936700 10 mds.0.log _replay_thread kicking waiters
  145. 2013-04-09 02:22:54.494993 7f5878936700 1 mds.0.35 replay_done
  146. 2013-04-09 02:22:54.494999 7f5878936700 1 mds.0.35 making mds journal writeable
  147. 2013-04-09 02:22:54.497933 7f5878936700 2 mds.0.35 i am alone, moving to state reconnect
  148. 2013-04-09 02:22:54.498162 7f5878936700 3 mds.0.35 request_state up:reconnect
  149. 2013-04-09 02:22:54.498264 7f5878936700 10 mds.0.35 beacon_send up:reconnect seq 11 (currently up:replay)
  150. 2013-04-09 02:22:54.498421 7f5878936700 10 mds.0.log _replay_thread finish
  151. 2013-04-09 02:22:55.530336 7f587b640700 5 mds.0.35 handle_mds_map epoch 793 from mon.0
  152. 2013-04-09 02:22:55.530361 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  153. 2013-04-09 02:22:55.530365 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  154. 2013-04-09 02:22:55.530367 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:reconnect
  155. 2013-04-09 02:22:55.530370 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  156. 2013-04-09 02:22:55.530371 7f587b640700 1 mds.0.35 handle_mds_map state change up:replay --> up:reconnect
  157. 2013-04-09 02:22:55.530373 7f587b640700 1 mds.0.35 reconnect_start
  158. 2013-04-09 02:22:55.530374 7f587b640700 1 mds.0.35 reopen_log
  159. 2013-04-09 02:22:55.530375 7f587b640700 10 mds.0.cache rollback_uncommitted_fragments: 0 pending
  160. 2013-04-09 02:22:55.530384 7f587b640700 7 mds.0.server reconnect_clients -- no sessions, doing nothing.
  161. 2013-04-09 02:22:55.530394 7f587b640700 7 mds.0.server reconnect_gather_finish. failed on 0 clients
  162. 2013-04-09 02:22:55.530395 7f587b640700 1 mds.0.35 reconnect_done
  163. 2013-04-09 02:22:55.530396 7f587b640700 3 mds.0.35 request_state up:rejoin
  164. 2013-04-09 02:22:55.530397 7f587b640700 10 mds.0.35 beacon_send up:rejoin seq 12 (currently up:reconnect)
  165. 2013-04-09 02:22:55.530422 7f587b640700 10 mds.0.bal check_targets have need want
  166. 2013-04-09 02:22:55.530461 7f587b640700 15 mds.0.bal map: i imported [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] from 0
  167. 2013-04-09 02:22:55.530468 7f587b640700 5 mds.0.bal rebalance done
  168. 2013-04-09 02:22:55.530470 7f587b640700 15 mds.0.cache show_subtrees
  169. 2013-04-09 02:22:55.530474 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  170. 2013-04-09 02:22:56.049360 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  171. 2013-04-09 02:22:56.468064 7f587b640700 5 mds.0.35 handle_mds_map epoch 794 from mon.0
  172. 2013-04-09 02:22:56.468082 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  173. 2013-04-09 02:22:56.468085 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  174. 2013-04-09 02:22:56.468088 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:rejoin
  175. 2013-04-09 02:22:56.468092 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  176. 2013-04-09 02:22:56.468093 7f587b640700 1 mds.0.35 handle_mds_map state change up:reconnect --> up:rejoin
  177. 2013-04-09 02:22:56.468095 7f587b640700 1 mds.0.35 rejoin_joint_start
  178. 2013-04-09 02:22:56.468104 7f587b640700 10 mds.0.cache rejoin_send_rejoins with recovery_set
  179. 2013-04-09 02:22:56.468122 7f587b640700 10 mds.0.cache nothing to rejoin
  180. 2013-04-09 02:22:56.468125 7f587b640700 10 mds.0.cache rejoin_gather_finish
  181. 2013-04-09 02:22:56.468128 7f587b640700 10 mds.0.cache rejoin_trim_undef_inodes
  182. 2013-04-09 02:22:56.468131 7f587b640700 10 mds.0.cache process_imported_caps
  183. 2013-04-09 02:22:56.468141 7f587b640700 10 mds.0.cache choose_lock_states_and_reconnect_caps
  184. 2013-04-09 02:22:56.468147 7f587b640700 15 mds.0.cache chose lock states on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  185. 2013-04-09 02:22:56.468163 7f587b640700 10 mds.0.cache.snaprealm(1 seq 1 0x7f587401b280) have_past_parents_open [1,head]
  186. 2013-04-09 02:22:56.468165 7f587b640700 10 mds.0.cache have past snap parents for realm snaprealm(1 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f587401b280) on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  187. 2013-04-09 02:22:56.468171 7f587b640700 15 mds.0.cache chose lock states on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  188. 2013-04-09 02:22:56.468175 7f587b640700 10 mds.0.cache.snaprealm(100 seq 1 0x7f5874018d20) have_past_parents_open [1,head]
  189. 2013-04-09 02:22:56.468177 7f587b640700 10 mds.0.cache have past snap parents for realm snaprealm(100 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f5874018d20) on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  190. 2013-04-09 02:22:56.468183 7f587b640700 10 mds.0.cache send_snaps
  191. 2013-04-09 02:22:56.468184 7f587b640700 10 mds.0.cache identify_files_to_recover
  192. 2013-04-09 02:22:56.468189 7f587b640700 7 mds.0.cache rejoin_send_acks
  193. 2013-04-09 02:22:56.468190 7f587b640700 10 mds.0.cache subtree [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  194. 2013-04-09 02:22:56.468198 7f587b640700 10 mds.0.cache open_snap_parents
  195. 2013-04-09 02:22:56.468199 7f587b640700 10 mds.0.cache send_snaps
  196. 2013-04-09 02:22:56.468200 7f587b640700 10 mds.0.cache open_snap_parents - all open
  197. 2013-04-09 02:22:56.468201 7f587b640700 10 mds.0.cache do_delayed_cap_imports
  198. 2013-04-09 02:22:56.468203 7f587b640700 10 mds.0.cache open_undef_dirfrags 0 dirfrags
  199. 2013-04-09 02:22:56.468208 7f587b640700 20 mds.0.locker check_inode_max_size no-op on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  200. 2013-04-09 02:22:56.468217 7f587b640700 20 mds.0.locker check_inode_max_size no-op on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  201. 2013-04-09 02:22:56.468221 7f587b640700 1 mds.0.35 rejoin_done
  202. 2013-04-09 02:22:56.468223 7f587b640700 15 mds.0.cache show_subtrees
  203. 2013-04-09 02:22:56.468227 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  204. 2013-04-09 02:22:56.468232 7f587b640700 7 mds.0.cache show_cache
  205. 2013-04-09 02:22:56.468233 7f587b640700 7 mds.0.cache unlinked [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  206. 2013-04-09 02:22:56.468238 7f587b640700 7 mds.0.cache dirfrag [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  207. 2013-04-09 02:22:56.468241 7f587b640700 7 mds.0.cache unlinked [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  208. 2013-04-09 02:22:56.468245 7f587b640700 3 mds.0.35 request_state up:active
  209. 2013-04-09 02:22:56.468246 7f587b640700 10 mds.0.35 beacon_send up:active seq 13 (currently up:rejoin)
  210. 2013-04-09 02:22:56.468283 7f587b640700 10 mds.0.bal check_targets have need want
  211. 2013-04-09 02:22:56.468287 7f587b640700 15 mds.0.bal map: i imported [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] from 0
  212. 2013-04-09 02:22:56.468290 7f587b640700 5 mds.0.bal rebalance done
  213. 2013-04-09 02:22:56.468292 7f587b640700 15 mds.0.cache show_subtrees
  214. 2013-04-09 02:22:56.468294 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  215. 2013-04-09 02:22:58.577418 7f587b640700 5 mds.0.35 handle_mds_map epoch 795 from mon.0
  216. 2013-04-09 02:22:58.577457 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  217. 2013-04-09 02:22:58.577467 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  218. 2013-04-09 02:22:58.577475 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:active
  219. 2013-04-09 02:22:58.577485 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  220. 2013-04-09 02:22:58.577489 7f587b640700 1 mds.0.35 handle_mds_map state change up:rejoin --> up:active
  221. 2013-04-09 02:22:58.577494 7f587b640700 1 mds.0.35 recovery_done -- successful recovery!
  222. 2013-04-09 02:22:58.577507 7f587b640700 7 mds.0.tableserver(anchortable) finish_recovery
  223. 2013-04-09 02:22:58.577515 7f587b640700 7 mds.0.tableserver(snaptable) finish_recovery
  224. 2013-04-09 02:22:58.577521 7f587b640700 7 mds.0.tableclient(anchortable) finish_recovery
  225. 2013-04-09 02:22:58.577525 7f587b640700 7 mds.0.tableclient(snaptable) finish_recovery
  226. 2013-04-09 02:22:58.577529 7f587b640700 10 mds.0.cache start_recovered_truncates
  227. 2013-04-09 02:22:58.577533 7f587b640700 10 mds.0.cache do_file_recover 0 queued, 0 recovering
  228. 2013-04-09 02:22:58.577541 7f587b640700 10 mds.0.cache reissue_all_caps
  229. 2013-04-09 02:22:58.581855 7f587b640700 -1 mds/MDCache.cc: In function 'void MDCache::populate_mydir()' thread 7f587b640700 time 2013-04-09 02:22:58.577558
  230. mds/MDCache.cc: 579: FAILED assert(mydir)
  231.  
  232. ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
  233. 1: (MDCache::populate_mydir()+0xbc5) [0x5f0125]
  234. 2: (MDS::recovery_done()+0xde) [0x4ed12e]
  235. 3: (MDS::handle_mds_map(MMDSMap*)+0x39c8) [0x4fff28]
  236. 4: (MDS::handle_core_message(Message*)+0xb4b) [0x50596b]
  237. 5: (MDS::_dispatch(Message*)+0x2f) [0x505a9f]
  238. 6: (MDS::ms_dispatch(Message*)+0x23b) [0x50759b]
  239. 7: (Messenger::ms_deliver_dispatch(Message*)+0x66) [0x872a26]
  240. 8: (DispatchQueue::entry()+0x32a) [0x87093a]
  241. 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ee7cd]
  242. 10: (()+0x6a3f) [0x7f587f465a3f]
  243. 11: (clone()+0x6d) [0x7f587df1967d]
  244. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  245.  
  246. --- begin dump of recent events ---
  247. -390> 2013-04-09 02:22:16.042446 7f587f88a760 5 asok(0x131ea40) register_command perfcounters_dump hook 0x131fbf0
  248. -389> 2013-04-09 02:22:16.042472 7f587f88a760 5 asok(0x131ea40) register_command 1 hook 0x131fbf0
  249. -388> 2013-04-09 02:22:16.042474 7f587f88a760 5 asok(0x131ea40) register_command perf dump hook 0x131fbf0
  250. -387> 2013-04-09 02:22:16.042483 7f587f88a760 5 asok(0x131ea40) register_command perfcounters_schema hook 0x131fbf0
  251. -386> 2013-04-09 02:22:16.042493 7f587f88a760 5 asok(0x131ea40) register_command 2 hook 0x131fbf0
  252. -385> 2013-04-09 02:22:16.042495 7f587f88a760 5 asok(0x131ea40) register_command perf schema hook 0x131fbf0
  253. -384> 2013-04-09 02:22:16.042497 7f587f88a760 5 asok(0x131ea40) register_command config show hook 0x131fbf0
  254. -383> 2013-04-09 02:22:16.042499 7f587f88a760 5 asok(0x131ea40) register_command config set hook 0x131fbf0
  255. -382> 2013-04-09 02:22:16.042509 7f587f88a760 5 asok(0x131ea40) register_command log flush hook 0x131fbf0
  256. -381> 2013-04-09 02:22:16.042511 7f587f88a760 5 asok(0x131ea40) register_command log dump hook 0x131fbf0
  257. -380> 2013-04-09 02:22:16.042513 7f587f88a760 5 asok(0x131ea40) register_command log reopen hook 0x131fbf0
  258. -379> 2013-04-09 02:22:16.043551 7f587f88a760 0 ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061), process ceph-mds, pid 11967
  259. -378> 2013-04-09 02:22:16.043728 7f587f88a760 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/11967 need_addr=1
  260. -377> 2013-04-09 02:22:16.044175 7f587f88a760 1 finished global_init_daemonize
  261. -376> 2013-04-09 02:22:16.044284 7f587f88a760 5 asok(0x131ea40) init /var/run/ceph/ceph-mds.a.asok
  262. -375> 2013-04-09 02:22:16.044300 7f587f88a760 5 asok(0x131ea40) bind_and_listen /var/run/ceph/ceph-mds.a.asok
  263. -374> 2013-04-09 02:22:16.044325 7f587f88a760 5 asok(0x131ea40) register_command 0 hook 0x1322a90
  264. -373> 2013-04-09 02:22:16.044334 7f587f88a760 5 asok(0x131ea40) register_command version hook 0x1322a90
  265. -372> 2013-04-09 02:22:16.044341 7f587f88a760 5 asok(0x131ea40) register_command git_version hook 0x1322a90
  266. -371> 2013-04-09 02:22:16.044345 7f587f88a760 5 asok(0x131ea40) register_command help hook 0x1322980
  267. -370> 2013-04-09 02:22:16.044386 7f587f88a760 10 monclient(hunting): build_initial_monmap
  268. -369> 2013-04-09 02:22:16.044427 7f587ce43700 5 asok(0x131ea40) entry start
  269. -368> 2013-04-09 02:22:16.044444 7f587f88a760 1 -- 0.0.0.0:6800/11967 messenger.start
  270. -367> 2013-04-09 02:22:16.044630 7f587f88a760 5 adding auth protocol: none
  271. -366> 2013-04-09 02:22:16.044670 7f587f88a760 5 adding auth protocol: none
  272. -365> 2013-04-09 02:22:16.044730 7f587f88a760 10 mds.-1.0 168 MDSCacheObject
  273. -364> 2013-04-09 02:22:16.044809 7f587f88a760 10 mds.-1.0 2144 CInode
  274. -363> 2013-04-09 02:22:16.044825 7f587f88a760 10 mds.-1.0 16 elist<>::item *7=112
  275. -362> 2013-04-09 02:22:16.044838 7f587f88a760 10 mds.-1.0 360 inode_t
  276. -361> 2013-04-09 02:22:16.045432 7f587f88a760 10 mds.-1.0 56 nest_info_t
  277. -360> 2013-04-09 02:22:16.045450 7f587f88a760 10 mds.-1.0 32 frag_info_t
  278. -359> 2013-04-09 02:22:16.045463 7f587f88a760 10 mds.-1.0 40 SimpleLock *5=200
  279. -358> 2013-04-09 02:22:16.045479 7f587f88a760 10 mds.-1.0 48 ScatterLock *3=144
  280. -357> 2013-04-09 02:22:16.045492 7f587f88a760 10 mds.-1.0 472 CDentry
  281. -356> 2013-04-09 02:22:16.045507 7f587f88a760 10 mds.-1.0 16 elist<>::item
  282. -355> 2013-04-09 02:22:16.045520 7f587f88a760 10 mds.-1.0 40 SimpleLock
  283. -354> 2013-04-09 02:22:16.045533 7f587f88a760 10 mds.-1.0 1016 CDir
  284. -353> 2013-04-09 02:22:16.045545 7f587f88a760 10 mds.-1.0 16 elist<>::item *2=32
  285. -352> 2013-04-09 02:22:16.045558 7f587f88a760 10 mds.-1.0 192 fnode_t
  286. -351> 2013-04-09 02:22:16.045571 7f587f88a760 10 mds.-1.0 56 nest_info_t *2
  287. -350> 2013-04-09 02:22:16.045583 7f587f88a760 10 mds.-1.0 32 frag_info_t *2
  288. -349> 2013-04-09 02:22:16.045596 7f587f88a760 10 mds.-1.0 168 Capability
  289. -348> 2013-04-09 02:22:16.045609 7f587f88a760 10 mds.-1.0 32 xlist<>::item *2=64
  290. -347> 2013-04-09 02:22:16.045654 7f587f88a760 1 accepter.accepter.start
  291. -346> 2013-04-09 02:22:16.045700 7f587f88a760 10 monclient(hunting): init
  292. -345> 2013-04-09 02:22:16.045711 7f587f88a760 5 adding auth protocol: none
  293. -344> 2013-04-09 02:22:16.045718 7f587f88a760 10 monclient(hunting): auth_supported 1 method none
  294. -343> 2013-04-09 02:22:16.045817 7f587f88a760 10 monclient(hunting): _reopen_session
  295. -342> 2013-04-09 02:22:16.045872 7f587f88a760 10 monclient(hunting): _pick_new_mon picked mon.a con 0x132c740 addr mon.mon.mon.mon:6789/0
  296. -341> 2013-04-09 02:22:16.045914 7f587f88a760 10 monclient(hunting): _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  297. -340> 2013-04-09 02:22:16.045929 7f587f88a760 1 -- 0.0.0.0:6800/11967 --> mon.mon.mon.mon:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x132cc40 con 0x132c740
  298. -339> 2013-04-09 02:22:16.045953 7f587f88a760 10 monclient(hunting): renew_subs
  299. -338> 2013-04-09 02:22:16.046340 7f587f886700 1 -- mds.mds.mds.mds:6800/11967 learned my addr mds.mds.mds.mds:6800/11967
  300. -337> 2013-04-09 02:22:16.046417 7f587f886700 10 mds.-1.0 MDS::ms_get_authorizer type=mon
  301. -336> 2013-04-09 02:22:16.046737 7f587b640700 0 mds.-1.0 ms_handle_connect on mon.mon.mon.mon:6789/0
  302. -335> 2013-04-09 02:22:16.047505 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 1 ==== mon_map v1 ==== 191+0+0 (2111496365 0 0) 0x132cc40 con 0x132c740
  303. -334> 2013-04-09 02:22:16.047558 7f587b640700 10 monclient(hunting): handle_monmap mon_map v1
  304. -333> 2013-04-09 02:22:16.047571 7f587b640700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
  305. -332> 2013-04-09 02:22:16.047574 7f587b640700 10 monclient(hunting): dump:
  306. epoch 1
  307. fsid 0ead9ea2-d9f4-4a14-ad9f-c0c785884511
  308. last_changed 2013-03-31 05:05:19.195480
  309. created 2013-03-31 05:05:19.195480
  310. 0: mon.mon.mon.mon:6789/0 mon.a
  311.  
  312. -331> 2013-04-09 02:22:16.047646 7f587b640700 1 monclient(hunting): found mon.a
  313. -330> 2013-04-09 02:22:16.047652 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 2 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (1550453269 0 0) 0x132d1a0 con 0x132c740
  314. -329> 2013-04-09 02:22:16.047686 7f587b640700 10 monclient: my global_id is 7697
  315. -328> 2013-04-09 02:22:16.047689 7f587b640700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  316. -327> 2013-04-09 02:22:16.047708 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x132e1b0 con 0x132c740
  317. -326> 2013-04-09 02:22:16.047742 7f587b640700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-09 02:21:46.047740)
  318. -325> 2013-04-09 02:22:16.047831 7f587f88a760 5 monclient: authenticate success, global_id 7697
  319. -324> 2013-04-09 02:22:16.047869 7f587f88a760 5 asok(0x131ea40) register_command objecter_requests hook 0x132d820
  320. -323> 2013-04-09 02:22:16.047937 7f587f88a760 10 mds.-1.0 beacon_send up:boot seq 1 (currently up:boot)
  321. -322> 2013-04-09 02:22:16.047963 7f587f88a760 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  322. -321> 2013-04-09 02:22:16.047968 7f587f88a760 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:boot seq 1 v0) v2 -- ?+0 0x132daa0 con 0x132c740
  323. -320> 2013-04-09 02:22:16.047985 7f587f88a760 10 monclient: renew_subs
  324. -319> 2013-04-09 02:22:16.047998 7f587f88a760 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  325. -318> 2013-04-09 02:22:16.048001 7f587f88a760 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x132f600 con 0x132c740
  326. -317> 2013-04-09 02:22:16.048019 7f587f88a760 10 monclient: renew_subs
  327. -316> 2013-04-09 02:22:16.048021 7f587f88a760 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  328. -315> 2013-04-09 02:22:16.048024 7f587f88a760 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x132fb80 con 0x132c740
  329. -314> 2013-04-09 02:22:16.048041 7f587f88a760 10 mds.-1.0 create_logger
  330. -313> 2013-04-09 02:22:16.259143 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 3 ==== mon_map v1 ==== 191+0+0 (2111496365 0 0) 0x132f600 con 0x132c740
  331. -312> 2013-04-09 02:22:16.259178 7f587b640700 10 monclient: handle_monmap mon_map v1
  332. -311> 2013-04-09 02:22:16.259196 7f587b640700 10 monclient: got monmap 1, mon.a is now rank 0
  333. -310> 2013-04-09 02:22:16.259201 7f587b640700 10 monclient: dump:
  334. epoch 1
  335. fsid 0ead9ea2-d9f4-4a14-ad9f-c0c785884511
  336. last_changed 2013-03-31 05:05:19.195480
  337. created 2013-03-31 05:05:19.195480
  338. 0: mon.mon.mon.mon:6789/0 mon.a
  339.  
  340. -309> 2013-04-09 02:22:16.259226 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3963790625 0 0) 0x132db70 con 0x132c740
  341. -308> 2013-04-09 02:22:16.259239 7f587b640700 10 monclient: handle_subscribe_ack sent 2013-04-09 02:22:16.045958 renew after 2013-04-09 02:24:46.045958
  342. -307> 2013-04-09 02:22:16.427151 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 5 ==== osd_map(212..212 src has 1..212) v3 ==== 2523+0+0 (517645052 0 0) 0x1331c60 con 0x132c740
  343. -306> 2013-04-09 02:22:16.427323 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3963790625 0 0) 0x132db70 con 0x132c740
  344. -305> 2013-04-09 02:22:16.427366 7f587b640700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  345. -304> 2013-04-09 02:22:16.427379 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 7 ==== mdsmap(e 791) v1 ==== 733+0+0 (3437839338 0 0) 0x1332160 con 0x132c740
  346. -303> 2013-04-09 02:22:16.427405 7f587b640700 5 mds.-1.0 handle_mds_map epoch 791 from mon.0
  347. -302> 2013-04-09 02:22:16.427490 7f587b640700 10 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  348. -301> 2013-04-09 02:22:16.427516 7f587b640700 10 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  349. -300> 2013-04-09 02:22:16.427530 7f587b640700 10 mds.-1.0 map says i am mds.mds.mds.mds:6800/11967 mds.-1.0 state up:standby
  350. -299> 2013-04-09 02:22:16.427549 7f587b640700 1 mds.-1.0 handle_mds_map standby
  351. -298> 2013-04-09 02:22:16.427556 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 8 ==== osd_map(212..212 src has 1..212) v3 ==== 2523+0+0 (517645052 0 0) 0x1332e70 con 0x132c740
  352. -297> 2013-04-09 02:22:16.427569 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3963790625 0 0) 0x13331d0 con 0x132c740
  353. -296> 2013-04-09 02:22:16.427576 7f587b640700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
  354. -295> 2013-04-09 02:22:19.045962 7f587a63e700 10 monclient: tick
  355. -294> 2013-04-09 02:22:19.046021 7f587a63e700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-09 02:21:49.046018)
  356. -293> 2013-04-09 02:22:19.046082 7f587a63e700 10 monclient: renew subs? (now: 2013-04-09 02:22:19.046081; renew after: 2013-04-09 02:24:46.045958) -- no
  357. -292> 2013-04-09 02:22:20.048147 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 2 (currently up:standby)
  358. -291> 2013-04-09 02:22:20.048218 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  359. -290> 2013-04-09 02:22:20.048237 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 2 v791) v2 -- ?+0 0x1331a90 con 0x132c740
  360. -289> 2013-04-09 02:22:20.049318 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 10 ==== mdsbeacon(7697/a up:standby seq 2 v791) v2 ==== 103+0+0 (2722855582 0 0) 0x1331a90 con 0x132c740
  361. -288> 2013-04-09 02:22:20.049368 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 2 rtt 0.001157
  362. -287> 2013-04-09 02:22:21.048153 7f587953b700 20 mds.-1.bal get_load no root, no load
  363. -286> 2013-04-09 02:22:21.048442 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  364. -285> 2013-04-09 02:22:24.048497 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 3 (currently up:standby)
  365. -284> 2013-04-09 02:22:24.048582 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  366. -283> 2013-04-09 02:22:24.048592 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 3 v791) v2 -- ?+0 0x1331a90 con 0x132c740
  367. -282> 2013-04-09 02:22:24.049360 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 11 ==== mdsbeacon(7697/a up:standby seq 3 v791) v2 ==== 103+0+0 (3680634873 0 0) 0x1331a90 con 0x132c740
  368. -281> 2013-04-09 02:22:24.049420 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 3 rtt 0.000845
  369. -280> 2013-04-09 02:22:26.048306 7f587953b700 20 mds.-1.bal get_load no root, no load
  370. -279> 2013-04-09 02:22:26.048473 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  371. -278> 2013-04-09 02:22:28.048863 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 4 (currently up:standby)
  372. -277> 2013-04-09 02:22:28.048970 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  373. -276> 2013-04-09 02:22:28.048987 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 4 v791) v2 -- ?+0 0x13323a0 con 0x132c740
  374. -275> 2013-04-09 02:22:28.926083 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 12 ==== mdsbeacon(7697/a up:standby seq 4 v791) v2 ==== 103+0+0 (2974810941 0 0) 0x13323a0 con 0x132c740
  375. -274> 2013-04-09 02:22:28.926165 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 4 rtt 0.877204
  376. -273> 2013-04-09 02:22:29.046265 7f587a63e700 10 monclient: tick
  377. -272> 2013-04-09 02:22:29.046313 7f587a63e700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-09 02:21:59.046310)
  378. -271> 2013-04-09 02:22:29.046332 7f587a63e700 10 monclient: renew subs? (now: 2013-04-09 02:22:29.046332; renew after: 2013-04-09 02:24:46.045958) -- no
  379. -270> 2013-04-09 02:22:31.048447 7f587953b700 20 mds.-1.bal get_load no root, no load
  380. -269> 2013-04-09 02:22:31.048771 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  381. -268> 2013-04-09 02:22:32.049446 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 5 (currently up:standby)
  382. -267> 2013-04-09 02:22:32.049565 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  383. -266> 2013-04-09 02:22:32.049576 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 5 v791) v2 -- ?+0 0x1335000 con 0x132c740
  384. -265> 2013-04-09 02:22:32.702912 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 13 ==== mdsbeacon(7697/a up:standby seq 5 v791) v2 ==== 103+0+0 (3363405402 0 0) 0x1335000 con 0x132c740
  385. -264> 2013-04-09 02:22:32.702952 7f587b640700 10 mds.-1.0 handle_mds_beacon up:standby seq 5 rtt 0.653394
  386. -263> 2013-04-09 02:22:36.048607 7f587953b700 20 mds.-1.bal get_load no root, no load
  387. -262> 2013-04-09 02:22:36.048786 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  388. -261> 2013-04-09 02:22:36.049708 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 6 (currently up:standby)
  389. -260> 2013-04-09 02:22:36.049806 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  390. -259> 2013-04-09 02:22:36.049828 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 6 v791) v2 -- ?+0 0x1335250 con 0x132c740
  391. -258> 2013-04-09 02:22:39.046478 7f587a63e700 10 monclient: tick
  392. -257> 2013-04-09 02:22:39.046525 7f587a63e700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-09 02:22:09.046522)
  393. -256> 2013-04-09 02:22:39.046551 7f587a63e700 10 monclient: renew subs? (now: 2013-04-09 02:22:39.046550; renew after: 2013-04-09 02:24:46.045958) -- no
  394. -255> 2013-04-09 02:22:40.050077 7f587953b700 10 mds.-1.0 beacon_send up:standby seq 7 (currently up:standby)
  395. -254> 2013-04-09 02:22:40.050206 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  396. -253> 2013-04-09 02:22:40.050217 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:standby seq 7 v791) v2 -- ?+0 0x13358e0 con 0x132c740
  397. -252> 2013-04-09 02:22:41.048768 7f587953b700 20 mds.-1.bal get_load no root, no load
  398. -251> 2013-04-09 02:22:41.049024 7f587953b700 15 mds.-1.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  399. -250> 2013-04-09 02:22:41.874914 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 14 ==== mdsmap(e 792) v1 ==== 535+0+0 (3061580974 0 0) 0x1336000 con 0x132c740
  400. -249> 2013-04-09 02:22:41.874961 7f587b640700 5 mds.-1.0 handle_mds_map epoch 792 from mon.0
  401. -248> 2013-04-09 02:22:41.874989 7f587b640700 10 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  402. -247> 2013-04-09 02:22:41.874997 7f587b640700 10 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  403. -246> 2013-04-09 02:22:41.875003 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:replay
  404. -245> 2013-04-09 02:22:41.875012 7f587b640700 10 mds.0.35 peer mds gid 7597 removed from map
  405. -244> 2013-04-09 02:22:41.875016 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 mark_down mds.mds.mds.mds:6800/11657 -- pipe dne
  406. -243> 2013-04-09 02:22:41.875023 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  407. -242> 2013-04-09 02:22:41.875026 7f587b640700 1 mds.0.35 handle_mds_map state change up:standby --> up:replay
  408. -241> 2013-04-09 02:22:41.875038 7f587b640700 1 mds.0.35 replay_start
  409. -240> 2013-04-09 02:22:41.875079 7f587b640700 7 mds.0.cache set_recovery_set
  410. -239> 2013-04-09 02:22:41.875109 7f587b640700 1 mds.0.35 recovery set is
  411. -238> 2013-04-09 02:22:41.875113 7f587b640700 1 mds.0.35 need osdmap epoch 218, have 212
  412. -237> 2013-04-09 02:22:41.875117 7f587b640700 1 mds.0.35 waiting for osdmap 218 (which blacklists prior instance)
  413. -236> 2013-04-09 02:22:41.875131 7f587b640700 10 monclient: renew_subs
  414. -235> 2013-04-09 02:22:41.875139 7f587b640700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  415. -234> 2013-04-09 02:22:41.875153 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mon_subscribe({mdsmap=793+,monmap=2+,osdmap=213}) v2 -- ?+0 0x7f5874003410 con 0x132c740
  416. -233> 2013-04-09 02:22:41.875179 7f587b640700 7 mds.0.cache handle_mds_failure mds.0
  417. -232> 2013-04-09 02:22:41.875187 7f587b640700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are
  418. -231> 2013-04-09 02:22:41.875191 7f587b640700 10 mds.0.cache wants_resolve
  419. -230> 2013-04-09 02:22:41.875193 7f587b640700 10 mds.0.cache got_resolve
  420. -229> 2013-04-09 02:22:41.875195 7f587b640700 10 mds.0.cache rejoin_sent
  421. -228> 2013-04-09 02:22:41.875198 7f587b640700 10 mds.0.cache rejoin_gather
  422. -227> 2013-04-09 02:22:41.875200 7f587b640700 10 mds.0.cache rejoin_ack_gather
  423. -226> 2013-04-09 02:22:41.875206 7f587b640700 5 mds.0.migrator handle_mds_failure_or_stop mds.0
  424. -225> 2013-04-09 02:22:41.875242 7f587b640700 10 mds.0.cache show_subtrees - no subtrees
  425. -224> 2013-04-09 02:22:41.876310 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 15 ==== mdsbeacon(7697/a up:standby seq 6 v792) v2 ==== 103+0+0 (2383541975 0 0) 0x1331560 con 0x132c740
  426. -223> 2013-04-09 02:22:41.876347 7f587b640700 10 mds.0.35 handle_mds_beacon up:standby seq 6 rtt 5.826551
  427. -222> 2013-04-09 02:22:41.877500 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 16 ==== mdsbeacon(7697/a up:standby seq 7 v792) v2 ==== 103+0+0 (4147678128 0 0) 0x1331560 con 0x132c740
  428. -221> 2013-04-09 02:22:41.877544 7f587b640700 10 mds.0.35 handle_mds_beacon up:standby seq 7 rtt 1.827345
  429. -220> 2013-04-09 02:22:41.878460 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 17 ==== osd_map(213..218 src has 1..218) v3 ==== 1926+0+0 (3459332340 0 0) 0x1331560 con 0x132c740
  430. -219> 2013-04-09 02:22:41.878594 7f587b640700 2 mds.0.35 boot_start 1: opening inotable
  431. -218> 2013-04-09 02:22:41.878614 7f587b640700 10 mds.0.inotable: load
  432. -217> 2013-04-09 02:22:41.878857 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:1 mds0_inotable [read 0~0] 1.b852b893) v4 -- ?+0 0x7f5874005270 con 0x7f5874004f30
  433. -216> 2013-04-09 02:22:41.878898 7f587b640700 2 mds.0.35 boot_start 1: opening sessionmap
  434. -215> 2013-04-09 02:22:41.878908 7f587b640700 10 mds.0.sessionmap load
  435. -214> 2013-04-09 02:22:41.879233 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:2 mds0_sessionmap [read 0~0] 1.3270c60b) v4 -- ?+0 0x7f5874006340 con 0x7f5874005f80
  436. -213> 2013-04-09 02:22:41.879265 7f587b640700 2 mds.0.35 boot_start 1: opening anchor table
  437. -212> 2013-04-09 02:22:41.879273 7f587b640700 10 mds.0.anchortable: load
  438. -211> 2013-04-09 02:22:41.879309 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:3 mds_anchortable [read 0~0] 1.a977f6a7) v4 -- ?+0 0x7f5874006e70 con 0x7f5874005f80
  439. -210> 2013-04-09 02:22:41.879332 7f587b640700 2 mds.0.35 boot_start 1: opening snap table
  440. -209> 2013-04-09 02:22:41.879339 7f587b640700 10 mds.0.snaptable: load
  441. -208> 2013-04-09 02:22:41.879363 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:4 mds_snaptable [read 0~0] 1.d90270ad) v4 -- ?+0 0x1336720 con 0x7f5874005f80
  442. -207> 2013-04-09 02:22:41.879382 7f587b640700 2 mds.0.35 boot_start 1: opening mds log
  443. -206> 2013-04-09 02:22:41.879398 7f587b640700 5 mds.0.log open discovering log bounds
  444. -205> 2013-04-09 02:22:41.879408 7f587b640700 1 mds.0.journaler(ro) recover start
  445. -204> 2013-04-09 02:22:41.879414 7f587b640700 1 mds.0.journaler(ro) read_head
  446. -203> 2013-04-09 02:22:41.879471 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:5 200.00000000 [read 0~0] 1.844f3494) v4 -- ?+0 0x13374a0 con 0x7f5874004f30
  447. -202> 2013-04-09 02:22:41.879515 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 18 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3963790625 0 0) 0x1336570 con 0x132c740
  448. -201> 2013-04-09 02:22:41.879536 7f587b640700 10 monclient: handle_subscribe_ack sent 2013-04-09 02:22:41.875135 renew after 2013-04-09 02:25:11.875135
  449. -200> 2013-04-09 02:22:41.880582 7f5878d3a700 10 mds.0.35 MDS::ms_get_authorizer type=osd
  450. -199> 2013-04-09 02:22:41.881261 7f5878c39700 10 mds.0.35 MDS::ms_get_authorizer type=osd
  451. -198> 2013-04-09 02:22:41.881724 7f587b640700 0 mds.0.35 ms_handle_connect on osd0.ods0.osd0.osd0:6800/12654
  452. -197> 2013-04-09 02:22:41.881982 7f587b640700 0 mds.0.35 ms_handle_connect on osd1.ods1.osd1.osd1:6800/13459
  453. -196> 2013-04-09 02:22:41.953009 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 1 ==== osd_op_reply(5 200.00000000 [read 0~84] = 0) v4 ==== 111+0+84 (3966742553 0 1407514607) 0x1335fd0 con 0x7f5874004f30
  454. -195> 2013-04-09 02:22:41.953142 7f587b640700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 4194304, expire 4195106, write 4195106). probing for end of log (from 4195106)...
  455. -194> 2013-04-09 02:22:41.953155 7f587b640700 1 mds.0.journaler(ro) probing for end of the log
  456. -193> 2013-04-09 02:22:41.953210 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:6 200.00000001 [stat] 1.6e5f474 rwordered) v4 -- ?+0 0x7f5874006340 con 0x7f5874005f80
  457. -192> 2013-04-09 02:22:41.953250 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:7 200.00000002 [stat] 1.eb272dbb rwordered) v4 -- ?+0 0x7f5874007ae0 con 0x7f5874005f80
  458. -191> 2013-04-09 02:22:41.953549 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 2 ==== osd_op_reply(1 mds0_inotable [read 0~29] = 0) v4 ==== 112+0+29 (260568704 0 1403865816) 0x1335fd0 con 0x7f5874004f30
  459. -190> 2013-04-09 02:22:41.953572 7f587b640700 10 mds.0.inotable: load_2 got 29 bytes
  460. -189> 2013-04-09 02:22:41.953577 7f587b640700 10 mds.0.inotable: load_2 loaded v0
  461. -188> 2013-04-09 02:22:42.229217 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~17] = 0) v4 ==== 114+0+17 (2540721537 0 2334645356) 0x1335fd0 con 0x7f5874005f80
  462. -187> 2013-04-09 02:22:42.229260 7f587b640700 10 mds.0.sessionmap dump
  463. -186> 2013-04-09 02:22:42.229265 7f587b640700 10 mds.0.sessionmap _load_finish v 0, 0 sessions, 17 bytes
  464. -185> 2013-04-09 02:22:42.229267 7f587b640700 10 mds.0.sessionmap dump
  465. -184> 2013-04-09 02:22:42.654509 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 2 ==== osd_op_reply(3 mds_anchortable [read 0~29] = 0) v4 ==== 114+0+29 (2691427792 0 218791405) 0x1335fd0 con 0x7f5874005f80
  466. -183> 2013-04-09 02:22:42.654548 7f587b640700 10 mds.0.anchortable: load_2 got 29 bytes
  467. -182> 2013-04-09 02:22:42.654550 7f587b640700 10 mds.0.anchortable: load_2 loaded v0
  468. -181> 2013-04-09 02:22:43.170935 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 3 ==== osd_op_reply(4 mds_snaptable [read 0~41] = 0) v4 ==== 112+0+41 (2916466714 0 2312406677) 0x1335fd0 con 0x7f5874005f80
  469. -180> 2013-04-09 02:22:43.170973 7f587b640700 10 mds.0.snaptable: load_2 got 41 bytes
  470. -179> 2013-04-09 02:22:43.170975 7f587b640700 10 mds.0.snaptable: load_2 loaded v0
  471. -178> 2013-04-09 02:22:43.173708 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 4 ==== osd_op_reply(6 200.00000001 [stat] = 0) v4 ==== 111+0+16 (2139306498 0 3477594997) 0x1335fd0 con 0x7f5874005f80
  472. -177> 2013-04-09 02:22:43.176347 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 5 ==== osd_op_reply(7 200.00000002 [stat] = -2 (No such file or directory)) v4 ==== 111+0+0 (60251468 0 0) 0x1335fd0 con 0x7f5874005f80
  473. -176> 2013-04-09 02:22:43.176372 7f587b640700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 4195507 (header had 4195106). recovered.
  474. -175> 2013-04-09 02:22:43.176378 7f587b640700 2 mds.0.35 boot_start 2: loading/discovering base inodes
  475. -174> 2013-04-09 02:22:43.176383 7f587b640700 0 mds.0.cache creating system inode with ino:100
  476. -173> 2013-04-09 02:22:43.176432 7f587b640700 10 mds.0.cache.ino(100) fetch
  477. -172> 2013-04-09 02:22:43.176461 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:8 100.00000000 [getxattr inode] 1.c5265ab3) v4 -- ?+0 0x7f5874019330 con 0x7f5874004f30
  478. -171> 2013-04-09 02:22:43.176498 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:9 100.00000000.inode [read 0~0] 1.85dde07f) v4 -- ?+0 0x7f5874019c50 con 0x7f5874004f30
  479. -170> 2013-04-09 02:22:43.176512 7f587b640700 0 mds.0.cache creating system inode with ino:1
  480. -169> 2013-04-09 02:22:43.176517 7f587b640700 10 mds.0.cache.ino(1) fetch
  481. -168> 2013-04-09 02:22:43.176536 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:10 1.00000000 [getxattr inode] 1.6b2cdaff) v4 -- ?+0 0x7f587401b890 con 0x7f5874004f30
  482. -167> 2013-04-09 02:22:43.176581 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:11 1.00000000.inode [read 0~0] 1.232c0e14) v4 -- ?+0 0x7f5874019c50 con 0x7f5874004f30
  483. -166> 2013-04-09 02:22:43.445779 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 3 ==== osd_op_reply(8 100.00000000 [getxattr] = -61 (No data available)) v4 ==== 111+0+0 (2997825048 0 0) 0x1335fd0 con 0x7f5874004f30
  484. -165> 2013-04-09 02:22:43.503242 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 4 ==== osd_op_reply(9 100.00000000.inode [read 0~399] = 0) v4 ==== 117+0+399 (1291489066 0 1977849122) 0x13370d0 con 0x7f5874004f30
  485. -164> 2013-04-09 02:22:43.503350 7f587b640700 10 mds.0.cache.ino(100) _fetched got 0 and 399
  486. -163> 2013-04-09 02:22:43.503365 7f587b640700 10 mds.0.cache.ino(100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  487. -162> 2013-04-09 02:22:43.503393 7f587b640700 20 mds.0.cache.ino(100) decode_snap_blob snaprealm(100 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f5874018d20)
  488. -161> 2013-04-09 02:22:43.503410 7f587b640700 10 mds.0.cache.ino(100) _fetched [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  489. -160> 2013-04-09 02:22:44.050454 7f587953b700 10 mds.0.35 beacon_send up:replay seq 8 (currently up:replay)
  490. -159> 2013-04-09 02:22:44.050522 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  491. -158> 2013-04-09 02:22:44.050542 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:replay seq 8 v792) v2 -- ?+0 0x13370d0 con 0x132c740
  492. -157> 2013-04-09 02:22:44.573574 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 5 ==== osd_op_reply(10 1.00000000 [getxattr] = -61 (No data available)) v4 ==== 109+0+0 (4041465830 0 0) 0x1335fd0 con 0x7f5874004f30
  493. -156> 2013-04-09 02:22:44.577102 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 6 ==== osd_op_reply(11 1.00000000.inode [read 0~428] = 0) v4 ==== 115+0+428 (1340231224 0 3805918408) 0x13363e0 con 0x7f5874004f30
  494. -155> 2013-04-09 02:22:44.577154 7f587b640700 10 mds.0.cache.ino(1) _fetched got 0 and 428
  495. -154> 2013-04-09 02:22:44.577160 7f587b640700 10 mds.0.cache.ino(1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  496. -153> 2013-04-09 02:22:44.577169 7f587b640700 20 mds.0.cache.ino(1) decode_snap_blob snaprealm(1 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f587401b280)
  497. -152> 2013-04-09 02:22:44.577174 7f587b640700 10 mds.0.cache.ino(1) _fetched [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) 0x7f5874008960]
  498. -151> 2013-04-09 02:22:44.577184 7f587b640700 2 mds.0.35 boot_start 3: replaying mds log
  499. -150> 2013-04-09 02:22:44.577187 7f587b640700 10 mds.0.log replay start, from 4195106 to 4195507
  500. -149> 2013-04-09 02:22:44.577270 7f5878936700 10 mds.0.log _replay_thread start
  501. -148> 2013-04-09 02:22:44.577304 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:12 200.00000001 [read 802~401] 1.6e5f474) v4 -- ?+0 0x13363e0 con 0x7f5874005f80
  502. -147> 2013-04-09 02:22:45.526785 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 19 ==== mdsbeacon(7697/a up:replay seq 8 v792) v2 ==== 103+0+0 (1121010723 0 0) 0x13363e0 con 0x132c740
  503. -146> 2013-04-09 02:22:45.526865 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 8 rtt 1.476360
  504. -145> 2013-04-09 02:22:46.048991 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  505. -144> 2013-04-09 02:22:48.050750 7f587953b700 10 mds.0.35 beacon_send up:replay seq 9 (currently up:replay)
  506. -143> 2013-04-09 02:22:48.050814 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  507. -142> 2013-04-09 02:22:48.050829 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:replay seq 9 v792) v2 -- ?+0 0x13377b0 con 0x132c740
  508. -141> 2013-04-09 02:22:49.046754 7f587a63e700 10 monclient: tick
  509. -140> 2013-04-09 02:22:49.046784 7f587a63e700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-09 02:22:19.046782)
  510. -139> 2013-04-09 02:22:49.046804 7f587a63e700 10 monclient: renew subs? (now: 2013-04-09 02:22:49.046804; renew after: 2013-04-09 02:25:11.875135) -- no
  511. -138> 2013-04-09 02:22:49.410629 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 20 ==== mdsbeacon(7697/a up:replay seq 9 v792) v2 ==== 103+0+0 (1006123332 0 0) 0x1337a20 con 0x132c740
  512. -137> 2013-04-09 02:22:49.410729 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 9 rtt 1.359929
  513. -136> 2013-04-09 02:22:51.049179 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  514. -135> 2013-04-09 02:22:52.050963 7f587953b700 10 mds.0.35 beacon_send up:replay seq 10 (currently up:replay)
  515. -134> 2013-04-09 02:22:52.051012 7f587953b700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  516. -133> 2013-04-09 02:22:52.051021 7f587953b700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:replay seq 10 v792) v2 -- ?+0 0x1337af0 con 0x132c740
  517. -132> 2013-04-09 02:22:53.301482 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 21 ==== mdsbeacon(7697/a up:replay seq 10 v792) v2 ==== 103+0+0 (2961424109 0 0) 0x1337af0 con 0x132c740
  518. -131> 2013-04-09 02:22:53.301578 7f587b640700 10 mds.0.35 handle_mds_beacon up:replay seq 10 rtt 1.250575
  519. -130> 2013-04-09 02:22:54.494330 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.1 osd1.ods1.osd1.osd1:6800/13459 6 ==== osd_op_reply(12 200.00000001 [read 802~401] = 0) v4 ==== 111+0+401 (4238238146 0 597260546) 0x13381f0 con 0x7f5874005f80
  520. -129> 2013-04-09 02:22:54.494526 7f5878936700 10 mds.0.log _replay 4195106~397 / 4195507 2013-04-08 01:59:07.943387: ESubtreeMap 1 subtrees , 0 ambiguous [metablob 1, 1 dirs]
  521. -128> 2013-04-09 02:22:54.494598 7f5878936700 10 mds.0.journal ESubtreeMap.replay -- reconstructing (auth) subtree spanning tree
  522. -127> 2013-04-09 02:22:54.494613 7f5878936700 10 mds.0.journal EMetaBlob.replay 1 dirlumps by unknown.0
  523. -126> 2013-04-09 02:22:54.494619 7f5878936700 10 mds.0.journal EMetaBlob.replay dir 1
  524. -125> 2013-04-09 02:22:54.494631 7f5878936700 10 mds.0.cache force_dir_fragment * on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) 0x7f5874008960]
  525. -124> 2013-04-09 02:22:54.494664 7f5878936700 10 mds.0.cache force_dir_fragment no frags under *
  526. -123> 2013-04-09 02:22:54.494732 7f5878936700 7 mds.0.cache adjust_subtree_auth -1,-2 -> -2,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 0x1339d40]
  527. -122> 2013-04-09 02:22:54.494754 7f5878936700 10 mds.0.cache show_subtrees - no subtrees
  528. -121> 2013-04-09 02:22:54.494774 7f5878936700 7 mds.0.cache current root is [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  529. -120> 2013-04-09 02:22:54.494788 7f5878936700 10 mds.0.cache.dir(1) setting dir_auth=-2,-2 from -1,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  530. -119> 2013-04-09 02:22:54.494799 7f5878936700 10 mds.0.cache.dir(1) new subtree root, adjusting auth_pins
  531. -118> 2013-04-09 02:22:54.494807 7f5878936700 15 mds.0.cache show_subtrees
  532. -117> 2013-04-09 02:22:54.494817 7f5878936700 10 mds.0.cache |__-2 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  533. -116> 2013-04-09 02:22:54.494834 7f5878936700 10 mds.0.journal EMetaBlob.replay added dir [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  534. -115> 2013-04-09 02:22:54.494844 7f5878936700 10 mds.0.journal EMetaBlob.replay updated dir [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  535. -114> 2013-04-09 02:22:54.494868 7f5878936700 7 mds.0.cache adjust_bounded_subtree_auth -2,-2 -> 0,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] bound_dfs []
  536. -113> 2013-04-09 02:22:54.494886 7f5878936700 7 mds.0.cache adjust_bounded_subtree_auth -2,-2 -> 0,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] bounds
  537. -112> 2013-04-09 02:22:54.494904 7f5878936700 15 mds.0.cache show_subtrees
  538. -111> 2013-04-09 02:22:54.494913 7f5878936700 10 mds.0.cache |__-2 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  539. -110> 2013-04-09 02:22:54.494929 7f5878936700 7 mds.0.cache current root is [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  540. -109> 2013-04-09 02:22:54.494936 7f5878936700 10 mds.0.cache.dir(1) setting dir_auth=0,-2 from -2,-2 on [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=-2 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  541. -108> 2013-04-09 02:22:54.494948 7f5878936700 15 mds.0.cache show_subtrees
  542. -107> 2013-04-09 02:22:54.494952 7f5878936700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  543. -106> 2013-04-09 02:22:54.494960 7f5878936700 15 mds.0.cache show_subtrees
  544. -105> 2013-04-09 02:22:54.494963 7f5878936700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  545. -104> 2013-04-09 02:22:54.494978 7f5878936700 10 mds.0.log _replay - complete, 1 events
  546. -103> 2013-04-09 02:22:54.494983 7f5878936700 10 mds.0.log _replay_thread kicking waiters
  547. -102> 2013-04-09 02:22:54.494993 7f5878936700 1 mds.0.35 replay_done
  548. -101> 2013-04-09 02:22:54.494999 7f5878936700 1 mds.0.35 making mds journal writeable
  549. -100> 2013-04-09 02:22:54.495003 7f5878936700 1 mds.0.journaler(ro) set_writeable
  550. -99> 2013-04-09 02:22:54.497198 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:13 200.00000001 [zero 1203~4193101] 1.6e5f474) v4 -- ?+0 0x1342170 con 0x7f5874005f80
  551. -98> 2013-04-09 02:22:54.497385 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:14 200.00000002 [delete] 1.eb272dbb) v4 -- ?+0 0x1342640 con 0x7f5874005f80
  552. -97> 2013-04-09 02:22:54.497565 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:15 200.00000003 [delete] 1.95e3ab) v4 -- ?+0 0x1342a40 con 0x7f5874005f80
  553. -96> 2013-04-09 02:22:54.497746 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:16 200.00000004 [delete] 1.1ba6bc1f) v4 -- ?+0 0x1342e40 con 0x7f5874004f30
  554. -95> 2013-04-09 02:22:54.497832 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd0.ods0.osd0.osd0:6800/12654 -- osd_op(mds.0.35:17 200.00000005 [delete] 1.96d5827a) v4 -- ?+0 0x13434b0 con 0x7f5874004f30
  555. -94> 2013-04-09 02:22:54.497869 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> osd1.ods1.osd1.osd1:6800/13459 -- osd_op(mds.0.35:18 200.00000006 [delete] 1.ade7efde) v4 -- ?+0 0x1343be0 con 0x7f5874005f80
  556. -93> 2013-04-09 02:22:54.497933 7f5878936700 2 mds.0.35 i am alone, moving to state reconnect
  557. -92> 2013-04-09 02:22:54.498162 7f5878936700 3 mds.0.35 request_state up:reconnect
  558. -91> 2013-04-09 02:22:54.498264 7f5878936700 10 mds.0.35 beacon_send up:reconnect seq 11 (currently up:replay)
  559. -90> 2013-04-09 02:22:54.498369 7f5878936700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  560. -89> 2013-04-09 02:22:54.498387 7f5878936700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:reconnect seq 11 v792) v2 -- ?+0 0x7f587401ad10 con 0x132c740
  561. -88> 2013-04-09 02:22:54.498421 7f5878936700 10 mds.0.log _replay_thread finish
  562. -87> 2013-04-09 02:22:54.499683 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 7 ==== osd_op_reply(16 200.00000004 [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (3793709443 0 0) 0x1342e40 con 0x7f5874004f30
  563. -86> 2013-04-09 02:22:54.500159 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== osd.0 osd0.ods0.osd0.osd0:6800/12654 8 ==== osd_op_reply(17 200.00000005 [delete] ondisk = -2 (No such file or directory)) v4 ==== 111+0+0 (2923274340 0 0) 0x1342e40 con 0x7f5874004f30
  564. -85> 2013-04-09 02:22:55.530284 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 22 ==== mdsmap(e 793) v1 ==== 535+0+0 (289285137 0 0) 0x1342a40 con 0x132c740
  565. -84> 2013-04-09 02:22:55.530336 7f587b640700 5 mds.0.35 handle_mds_map epoch 793 from mon.0
  566. -83> 2013-04-09 02:22:55.530361 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  567. -82> 2013-04-09 02:22:55.530365 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  568. -81> 2013-04-09 02:22:55.530367 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:reconnect
  569. -80> 2013-04-09 02:22:55.530370 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  570. -79> 2013-04-09 02:22:55.530371 7f587b640700 1 mds.0.35 handle_mds_map state change up:replay --> up:reconnect
  571. -78> 2013-04-09 02:22:55.530373 7f587b640700 1 mds.0.35 reconnect_start
  572. -77> 2013-04-09 02:22:55.530374 7f587b640700 1 mds.0.35 reopen_log
  573. -76> 2013-04-09 02:22:55.530375 7f587b640700 10 mds.0.cache rollback_uncommitted_fragments: 0 pending
  574. -75> 2013-04-09 02:22:55.530384 7f587b640700 7 mds.0.server reconnect_clients -- no sessions, doing nothing.
  575. -74> 2013-04-09 02:22:55.530394 7f587b640700 7 mds.0.server reconnect_gather_finish. failed on 0 clients
  576. -73> 2013-04-09 02:22:55.530395 7f587b640700 1 mds.0.35 reconnect_done
  577. -72> 2013-04-09 02:22:55.530396 7f587b640700 3 mds.0.35 request_state up:rejoin
  578. -71> 2013-04-09 02:22:55.530397 7f587b640700 10 mds.0.35 beacon_send up:rejoin seq 12 (currently up:reconnect)
  579. -70> 2013-04-09 02:22:55.530403 7f587b640700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  580. -69> 2013-04-09 02:22:55.530406 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:rejoin seq 12 v793) v2 -- ?+0 0x7f587401d2e0 con 0x132c740
  581. -68> 2013-04-09 02:22:55.530422 7f587b640700 10 mds.0.bal check_targets have need want
  582. -67> 2013-04-09 02:22:55.530461 7f587b640700 15 mds.0.bal map: i imported [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] from 0
  583. -66> 2013-04-09 02:22:55.530468 7f587b640700 5 mds.0.bal rebalance done
  584. -65> 2013-04-09 02:22:55.530470 7f587b640700 15 mds.0.cache show_subtrees
  585. -64> 2013-04-09 02:22:55.530474 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  586. -63> 2013-04-09 02:22:56.049360 7f587953b700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
  587. -62> 2013-04-09 02:22:56.467999 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 23 ==== mdsmap(e 794) v1 ==== 535+0+0 (3617337845 0 0) 0x1343be0 con 0x132c740
  588. -61> 2013-04-09 02:22:56.468064 7f587b640700 5 mds.0.35 handle_mds_map epoch 794 from mon.0
  589. -60> 2013-04-09 02:22:56.468082 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  590. -59> 2013-04-09 02:22:56.468085 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  591. -58> 2013-04-09 02:22:56.468088 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:rejoin
  592. -57> 2013-04-09 02:22:56.468092 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  593. -56> 2013-04-09 02:22:56.468093 7f587b640700 1 mds.0.35 handle_mds_map state change up:reconnect --> up:rejoin
  594. -55> 2013-04-09 02:22:56.468095 7f587b640700 1 mds.0.35 rejoin_joint_start
  595. -54> 2013-04-09 02:22:56.468104 7f587b640700 10 mds.0.cache rejoin_send_rejoins with recovery_set
  596. -53> 2013-04-09 02:22:56.468122 7f587b640700 10 mds.0.cache nothing to rejoin
  597. -52> 2013-04-09 02:22:56.468125 7f587b640700 10 mds.0.cache rejoin_gather_finish
  598. -51> 2013-04-09 02:22:56.468128 7f587b640700 10 mds.0.cache rejoin_trim_undef_inodes
  599. -50> 2013-04-09 02:22:56.468131 7f587b640700 10 mds.0.cache process_imported_caps
  600. -49> 2013-04-09 02:22:56.468141 7f587b640700 10 mds.0.cache choose_lock_states_and_reconnect_caps
  601. -48> 2013-04-09 02:22:56.468147 7f587b640700 15 mds.0.cache chose lock states on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  602. -47> 2013-04-09 02:22:56.468163 7f587b640700 10 mds.0.cache.snaprealm(1 seq 1 0x7f587401b280) have_past_parents_open [1,head]
  603. -46> 2013-04-09 02:22:56.468165 7f587b640700 10 mds.0.cache have past snap parents for realm snaprealm(1 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f587401b280) on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  604. -45> 2013-04-09 02:22:56.468171 7f587b640700 15 mds.0.cache chose lock states on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  605. -44> 2013-04-09 02:22:56.468175 7f587b640700 10 mds.0.cache.snaprealm(100 seq 1 0x7f5874018d20) have_past_parents_open [1,head]
  606. -43> 2013-04-09 02:22:56.468177 7f587b640700 10 mds.0.cache have past snap parents for realm snaprealm(100 seq 1 lc 0 cr 0 cps 1 snaps={} 0x7f5874018d20) on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  607. -42> 2013-04-09 02:22:56.468183 7f587b640700 10 mds.0.cache send_snaps
  608. -41> 2013-04-09 02:22:56.468184 7f587b640700 10 mds.0.cache identify_files_to_recover
  609. -40> 2013-04-09 02:22:56.468189 7f587b640700 7 mds.0.cache rejoin_send_acks
  610. -39> 2013-04-09 02:22:56.468190 7f587b640700 10 mds.0.cache subtree [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  611. -38> 2013-04-09 02:22:56.468198 7f587b640700 10 mds.0.cache open_snap_parents
  612. -37> 2013-04-09 02:22:56.468199 7f587b640700 10 mds.0.cache send_snaps
  613. -36> 2013-04-09 02:22:56.468200 7f587b640700 10 mds.0.cache open_snap_parents - all open
  614. -35> 2013-04-09 02:22:56.468201 7f587b640700 10 mds.0.cache do_delayed_cap_imports
  615. -34> 2013-04-09 02:22:56.468203 7f587b640700 10 mds.0.cache open_undef_dirfrags 0 dirfrags
  616. -33> 2013-04-09 02:22:56.468208 7f587b640700 20 mds.0.locker check_inode_max_size no-op on [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  617. -32> 2013-04-09 02:22:56.468217 7f587b640700 20 mds.0.locker check_inode_max_size no-op on [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  618. -31> 2013-04-09 02:22:56.468221 7f587b640700 1 mds.0.35 rejoin_done
  619. -30> 2013-04-09 02:22:56.468223 7f587b640700 15 mds.0.cache show_subtrees
  620. -29> 2013-04-09 02:22:56.468227 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  621. -28> 2013-04-09 02:22:56.468232 7f587b640700 7 mds.0.cache show_cache
  622. -27> 2013-04-09 02:22:56.468233 7f587b640700 7 mds.0.cache unlinked [inode 1 [...2,head] / auth v1 snaprealm=0x7f587401b280 f(v0 1=0+1) n(v0 1=0+1) (iversion lock) | dirfrag 0x7f5874008960]
  623. -26> 2013-04-09 02:22:56.468238 7f587b640700 7 mds.0.cache dirfrag [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  624. -25> 2013-04-09 02:22:56.468241 7f587b640700 7 mds.0.cache unlinked [inode 100 [...2,head] ~mds0/ auth v1 snaprealm=0x7f5874018d20 f(v0 11=1+10) n(v0 11=1+10) (iversion lock) 0x7f5874008100]
  625. -24> 2013-04-09 02:22:56.468245 7f587b640700 3 mds.0.35 request_state up:active
  626. -23> 2013-04-09 02:22:56.468246 7f587b640700 10 mds.0.35 beacon_send up:active seq 13 (currently up:rejoin)
  627. -22> 2013-04-09 02:22:56.468258 7f587b640700 10 monclient: _send_mon_message to mon.a at mon.mon.mon.mon:6789/0
  628. -21> 2013-04-09 02:22:56.468264 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 --> mon.mon.mon.mon:6789/0 -- mdsbeacon(7697/a up:active seq 13 v794) v2 -- ?+0 0x7f5874020f80 con 0x132c740
  629. -20> 2013-04-09 02:22:56.468283 7f587b640700 10 mds.0.bal check_targets have need want
  630. -19> 2013-04-09 02:22:56.468287 7f587b640700 15 mds.0.bal map: i imported [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40] from 0
  631. -18> 2013-04-09 02:22:56.468290 7f587b640700 5 mds.0.bal rebalance done
  632. -17> 2013-04-09 02:22:56.468292 7f587b640700 15 mds.0.cache show_subtrees
  633. -16> 2013-04-09 02:22:56.468294 7f587b640700 10 mds.0.cache |__ 0 auth [dir 1 / [2,head] auth v=0 cv=0/0 dir_auth=0 state=1073741824 f() n() hs=0+0,ss=0+0 | subtree 0x1339d40]
  634. -15> 2013-04-09 02:22:58.577348 7f587b640700 1 -- mds.mds.mds.mds:6800/11967 <== mon.0 mon.mon.mon.mon:6789/0 24 ==== mdsmap(e 795) v1 ==== 535+0+0 (1955477328 0 0) 0x1343be0 con 0x132c740
  635. -14> 2013-04-09 02:22:58.577418 7f587b640700 5 mds.0.35 handle_mds_map epoch 795 from mon.0
  636. -13> 2013-04-09 02:22:58.577457 7f587b640700 10 mds.0.35 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  637. -12> 2013-04-09 02:22:58.577467 7f587b640700 10 mds.0.35 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
  638. -11> 2013-04-09 02:22:58.577475 7f587b640700 10 mds.0.35 map says i am mds.mds.mds.mds:6800/11967 mds.0.35 state up:active
  639. -10> 2013-04-09 02:22:58.577485 7f587b640700 1 mds.0.35 handle_mds_map i am now mds.0.35
  640. -9> 2013-04-09 02:22:58.577489 7f587b640700 1 mds.0.35 handle_mds_map state change up:rejoin --> up:active
  641. -8> 2013-04-09 02:22:58.577494 7f587b640700 1 mds.0.35 recovery_done -- successful recovery!
  642. -7> 2013-04-09 02:22:58.577507 7f587b640700 7 mds.0.tableserver(anchortable) finish_recovery
  643. -6> 2013-04-09 02:22:58.577515 7f587b640700 7 mds.0.tableserver(snaptable) finish_recovery
  644. -5> 2013-04-09 02:22:58.577521 7f587b640700 7 mds.0.tableclient(anchortable) finish_recovery
  645. -4> 2013-04-09 02:22:58.577525 7f587b640700 7 mds.0.tableclient(snaptable) finish_recovery
  646. -3> 2013-04-09 02:22:58.577529 7f587b640700 10 mds.0.cache start_recovered_truncates
  647. -2> 2013-04-09 02:22:58.577533 7f587b640700 10 mds.0.cache do_file_recover 0 queued, 0 recovering
  648. -1> 2013-04-09 02:22:58.577541 7f587b640700 10 mds.0.cache reissue_all_caps
  649. 0> 2013-04-09 02:22:58.581855 7f587b640700 -1 mds/MDCache.cc: In function 'void MDCache::populate_mydir()' thread 7f587b640700 time 2013-04-09 02:22:58.577558
  650. mds/MDCache.cc: 579: FAILED assert(mydir)
  651.  
  652. ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
  653. 1: (MDCache::populate_mydir()+0xbc5) [0x5f0125]
  654. 2: (MDS::recovery_done()+0xde) [0x4ed12e]
  655. 3: (MDS::handle_mds_map(MMDSMap*)+0x39c8) [0x4fff28]
  656. 4: (MDS::handle_core_message(Message*)+0xb4b) [0x50596b]
  657. 5: (MDS::_dispatch(Message*)+0x2f) [0x505a9f]
  658. 6: (MDS::ms_dispatch(Message*)+0x23b) [0x50759b]
  659. 7: (Messenger::ms_deliver_dispatch(Message*)+0x66) [0x872a26]
  660. 8: (DispatchQueue::entry()+0x32a) [0x87093a]
  661. 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ee7cd]
  662. 10: (()+0x6a3f) [0x7f587f465a3f]
  663. 11: (clone()+0x6d) [0x7f587df1967d]
  664. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  665.  
  666. --- logging levels ---
  667. 0/ 5 none
  668. 0/ 1 lockdep
  669. 0/ 1 context
  670. 1/ 1 crush
  671. 20/20 mds
  672. 1/ 5 mds_balancer
  673. 1/ 5 mds_locker
  674. 1/ 5 mds_log
  675. 1/ 5 mds_log_expire
  676. 1/ 5 mds_migrator
  677. 0/ 1 buffer
  678. 0/ 1 timer
  679. 0/ 1 filer
  680. 0/ 1 striper
  681. 0/ 1 objecter
  682. 0/ 5 rados
  683. 0/ 5 rbd
  684. 0/ 5 journaler
  685. 0/ 5 objectcacher
  686. 0/ 5 client
  687. 0/ 5 osd
  688. 0/ 5 optracker
  689. 0/ 5 objclass
  690. 1/ 3 filestore
  691. 1/ 3 journal
  692. 0/ 5 ms
  693. 1/ 5 mon
  694. 0/10 monc
  695. 0/ 5 paxos
  696. 0/ 5 tp
  697. 1/ 5 auth
  698. 1/ 5 crypto
  699. 1/ 1 finisher
  700. 1/ 5 heartbeatmap
  701. 1/ 5 perfcounter
  702. 1/ 5 rgw
  703. 1/ 5 hadoop
  704. 1/ 5 javaclient
  705. 1/ 5 asok
  706. 1/ 1 throttle
  707. -2/-2 (syslog threshold)
  708. -1/-1 (stderr threshold)
  709. max_recent 100000
  710. max_new 1000
  711. log_file /var/log/ceph/ceph-mds.a.log
  712. --- end dump of recent events ---
  713. 2013-04-09 02:22:58.597281 7f587b640700 -1 *** Caught signal (Aborted) **
  714. in thread 7f587b640700
  715.  
  716. ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
  717. 1: /usr/local/bin/ceph-mds() [0x89e4d9]
  718. 2: (()+0xf2d0) [0x7f587f46e2d0]
  719. 3: (gsignal()+0x35) [0x7f587de78ab5]
  720. 4: (abort()+0x186) [0x7f587de79fb6]
  721. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f587e6dea9d]
  722. 6: (()+0xbccb6) [0x7f587e6dccb6]
  723. 7: (()+0xbcce3) [0x7f587e6dcce3]
  724. 8: (()+0xbcdee) [0x7f587e6dcdee]
  725. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7a2) [0x800872]
  726. 10: (MDCache::populate_mydir()+0xbc5) [0x5f0125]
  727. 11: (MDS::recovery_done()+0xde) [0x4ed12e]
  728. 12: (MDS::handle_mds_map(MMDSMap*)+0x39c8) [0x4fff28]
  729. 13: (MDS::handle_core_message(Message*)+0xb4b) [0x50596b]
  730. 14: (MDS::_dispatch(Message*)+0x2f) [0x505a9f]
  731. 15: (MDS::ms_dispatch(Message*)+0x23b) [0x50759b]
  732. 16: (Messenger::ms_deliver_dispatch(Message*)+0x66) [0x872a26]
  733. 17: (DispatchQueue::entry()+0x32a) [0x87093a]
  734. 18: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ee7cd]
  735. 19: (()+0x6a3f) [0x7f587f465a3f]
  736. 20: (clone()+0x6d) [0x7f587df1967d]
  737. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  738.  
  739. --- begin dump of recent events ---
  740. 0> 2013-04-09 02:22:58.597281 7f587b640700 -1 *** Caught signal (Aborted) **
  741. in thread 7f587b640700
  742.  
  743. ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
  744. 1: /usr/local/bin/ceph-mds() [0x89e4d9]
  745. 2: (()+0xf2d0) [0x7f587f46e2d0]
  746. 3: (gsignal()+0x35) [0x7f587de78ab5]
  747. 4: (abort()+0x186) [0x7f587de79fb6]
  748. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f587e6dea9d]
  749. 6: (()+0xbccb6) [0x7f587e6dccb6]
  750. 7: (()+0xbcce3) [0x7f587e6dcce3]
  751. 8: (()+0xbcdee) [0x7f587e6dcdee]
  752. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7a2) [0x800872]
  753. 10: (MDCache::populate_mydir()+0xbc5) [0x5f0125]
  754. 11: (MDS::recovery_done()+0xde) [0x4ed12e]
  755. 12: (MDS::handle_mds_map(MMDSMap*)+0x39c8) [0x4fff28]
  756. 13: (MDS::handle_core_message(Message*)+0xb4b) [0x50596b]
  757. 14: (MDS::_dispatch(Message*)+0x2f) [0x505a9f]
  758. 15: (MDS::ms_dispatch(Message*)+0x23b) [0x50759b]
  759. 16: (Messenger::ms_deliver_dispatch(Message*)+0x66) [0x872a26]
  760. 17: (DispatchQueue::entry()+0x32a) [0x87093a]
  761. 18: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ee7cd]
  762. 19: (()+0x6a3f) [0x7f587f465a3f]
  763. 20: (clone()+0x6d) [0x7f587df1967d]
  764. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  765.  
  766. --- logging levels ---
  767. 0/ 5 none
  768. 0/ 1 lockdep
  769. 0/ 1 context
  770. 1/ 1 crush
  771. 20/20 mds
  772. 1/ 5 mds_balancer
  773. 1/ 5 mds_locker
  774. 1/ 5 mds_log
  775. 1/ 5 mds_log_expire
  776. 1/ 5 mds_migrator
  777. 0/ 1 buffer
  778. 0/ 1 timer
  779. 0/ 1 filer
  780. 0/ 1 striper
  781. 0/ 1 objecter
  782. 0/ 5 rados
  783. 0/ 5 rbd
  784. 0/ 5 journaler
  785. 0/ 5 objectcacher
  786. 0/ 5 client
  787. 0/ 5 osd
  788. 0/ 5 optracker
  789. 0/ 5 objclass
  790. 1/ 3 filestore
  791. 1/ 3 journal
  792. 0/ 5 ms
  793. 1/ 5 mon
  794. 0/10 monc
  795. 0/ 5 paxos
  796. 0/ 5 tp
  797. 1/ 5 auth
  798. 1/ 5 crypto
  799. 1/ 1 finisher
  800. 1/ 5 heartbeatmap
  801. 1/ 5 perfcounter
  802. 1/ 5 rgw
  803. 1/ 5 hadoop
  804. 1/ 5 javaclient
  805. 1/ 5 asok
  806. 1/ 1 throttle
  807. -2/-2 (syslog threshold)
  808. -1/-1 (stderr threshold)
  809. max_recent 100000
  810. max_new 1000
  811. log_file /var/log/ceph/ceph-mds.a.log
  812. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!