API tools faq
paste
Advertisement
Guest User

ceph mimic crash loop

a guest
Nov 6th, 2019
422
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 163.75 KB | None | 0 0
raw download clone embed print report
  1. 2019-11-06T08:17:09Z k8s-node-01 /tmp/tmp.bWsvKioECI/confd[2545]: INFO Backend set to etcd
  2. 2019-11-06T08:17:09Z k8s-node-01 /tmp/tmp.bWsvKioECI/confd[2545]: INFO Starting confd
  3. 2019-11-06T08:17:09Z k8s-node-01 /tmp/tmp.bWsvKioECI/confd[2545]: INFO Backend source(s) set to http://192.168.200.10:2379
  4. 2019-11-06 08:17:10 /entrypoint.sh: Adding bootstrap keyrings.
  5. 2019-11-06 08:17:11 /entrypoint.sh: Retrieving the admin key.
  6. 2019-11-06 08:17:13 /entrypoint.sh: SUCCESS
  7. 2019-11-06 08:17:13.577 7fa9b39d4000 0 set uid:gid to 167:167 (ceph:ceph)
  8. 2019-11-06 08:17:13.577 7fa9b39d4000 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mds, pid 2712
  9. 2019-11-06 08:17:13.577 7fa9b39d4000 0 pidfile_write: ignore empty --pid-file
  10. starting mds.k8s-node-01 at -
  11. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 112 MDSCacheObject
  12. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1696 CInode
  13. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *7=112
  14. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 488 inode
  15. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 504 old_inode
  16. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t
  17. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t
  18. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock *5=200
  19. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 ScatterLock *3=144
  20. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 480 CDentry
  21. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item
  22. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock
  23. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1600 CDir
  24. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *2=32
  25. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 232 fnode_t
  26. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t *2
  27. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t *2
  28. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 280 Capability
  29. 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 32 xlist<>::item *2=64
  30. 2019-11-06 08:17:13.661 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  31. 2019-11-06 08:17:13.665 7fa9a356a700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  32. 2019-11-06 08:17:13.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 1
  33. 2019-11-06 08:17:13.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  34. 2019-11-06 08:17:13.673 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26760 from mon.1
  35. 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  36. 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  37. 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.-1 state ???
  38. 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  39. 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 not in map yet
  40. 2019-11-06 08:17:13.673 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  41. 2019-11-06 08:17:14.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  42. 2019-11-06 08:17:15.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  43. 2019-11-06 08:17:16.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  44. 2019-11-06 08:17:17.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 2
  45. 2019-11-06 08:17:17.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  46. 2019-11-06 08:17:17.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  47. 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26761 from mon.1
  48. 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  49. 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  50. 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.0 state up:standby
  51. 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  52. 2019-11-06 08:17:18.605 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:boot -> up:standby
  53. 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Map has assigned me to become a standby
  54. 2019-11-06 08:17:18.605 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:boot seq 2 rtt 0.93202
  55. 2019-11-06 08:17:18.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  56. 2019-11-06 08:17:21.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 3
  57. 2019-11-06 08:17:21.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  58. 2019-11-06 08:17:21.677 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 3 rtt 0.00400009
  59. 2019-11-06 08:17:25.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 4
  60. 2019-11-06 08:17:25.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  61. 2019-11-06 08:17:25.709 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 4 rtt 0.0360008
  62. 2019-11-06 08:17:28.441 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26762 from mon.1
  63. 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  64. 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  65. 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.0.26762 state up:replay
  66. 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  67. 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: initializing MDS rank 0
  68. 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 update_log_config log_to_monitors {default=true}
  69. 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 create_logger
  70. 2019-11-06 08:17:28.445 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 0
  71. 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  72. 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries
  73. 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  74. 2019-11-06 08:17:28.449 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map as rank 0
  75. 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map i am now mds.0.26762
  76. 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map state change up:boot --> up:replay
  77. 2019-11-06 08:17:28.449 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:standby -> up:replay
  78. 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 replay_start
  79. 2019-11-06 08:17:28.449 7fa9a1566700 7 mds.0.cache set_recovery_set
  80. 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 recovery set is
  81. 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 waiting for osdmap 16073 (which blacklists prior instance)
  82. 2019-11-06 08:17:28.457 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 16073
  83. 2019-11-06 08:17:28.457 7fa9a1566700 4 mds.0.26762 handle_osd_map epoch 16073, 0 new blacklist entries
  84. 2019-11-06 08:17:28.457 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  85. 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_Wrapper
  86. 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  87. 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening inotable
  88. 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.inotable: load
  89. 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening sessionmap
  90. 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.sessionmap load
  91. 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening mds log
  92. 2019-11-06 08:17:28.461 7fa99b2d0700 5 mds.0.log open discovering log bounds
  93. 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening purge queue (async)
  94. 2019-11-06 08:17:28.461 7fa99b2d0700 4 mds.0.purge_queue open: opening
  95. 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: loading open file table (async)
  96. 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.openfiles load
  97. 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening snap table
  98. 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.snaptable: load
  99. 2019-11-06 08:17:28.461 7fa99a2ce700 10 mds.0.log _submit_thread start
  100. 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  101. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 got 70 bytes
  102. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 loaded v1
  103. 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: N12_GLOBAL__N_112C_IO_SM_LoadE
  104. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish loaded version 0
  105. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: omap load complete
  106. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: v 0, 0 sessions
  107. 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap dump
  108. 2019-11-06 08:17:28.485 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  109. 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 got 34 bytes
  110. 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 loaded v23241
  111. 2019-11-06 08:17:28.485 7fa99aacf700 4 mds.0.log Waiting for journal 0x200 to recover...
  112. 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Journal 0x200 recovered.
  113. 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Recovered journal 0x200 in format 1
  114. 2019-11-06 08:17:28.489 7fa99aacf700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  115. 2019-11-06 08:17:28.489 7fa99aacf700 2 mds.0.26762 boot_start 1: loading/discovering base inodes
  116. 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x100
  117. 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x100) fetch
  118. 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x1
  119. 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x1) fetch
  120. *** Caught signal (Segmentation fault) **
  121. in thread 7fa99b2d0700 thread_name:fn_anonymous
  122. 2019-11-06 08:17:28.493 7fa99b2d0700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
  123. 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482
  124. 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  125. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
  126. 1: (()+0xf5f0) [0x7fa9a87eb5f0]
  127. 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x55d74bca0e52]
  128. 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x55d74bca3648]
  129. 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x55d74bc56983]
  130. 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x43) [0x55d74bc773f3]
  131. 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x16a) [0x55d74bc775aa]
  132. 7: (MDSIOContextBase::complete(int)+0x119) [0x55d74bcc3679]
  133. 8: (Finisher::finisher_thread_entry()+0x12e) [0x7fa9aabae89e]
  134. 9: (()+0x7e65) [0x7fa9a87e3e65]
  135. 10: (clone()+0x6d) [0x7fa9a78be88d]
  136. 2019-11-06 08:17:28.497 7fa99b2d0700 -1 *** Caught signal (Segmentation fault) **
  137. in thread 7fa99b2d0700 thread_name:fn_anonymous
  138.  
  139. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
  140. 1: (()+0xf5f0) [0x7fa9a87eb5f0]
  141. 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x55d74bca0e52]
  142. 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x55d74bca3648]
  143. 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x55d74bc56983]
  144. 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x43) [0x55d74bc773f3]
  145. 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x16a) [0x55d74bc775aa]
  146. 7: (MDSIOContextBase::complete(int)+0x119) [0x55d74bcc3679]
  147. 8: (Finisher::finisher_thread_entry()+0x12e) [0x7fa9aabae89e]
  148. 9: (()+0x7e65) [0x7fa9a87e3e65]
  149. 10: (clone()+0x6d) [0x7fa9a78be88d]
  150. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  151.  
  152. --- begin dump of recent events ---
  153. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command abort hook 0x55d74cef62b0
  154. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perfcounters_dump hook 0x55d74cef62b0
  155. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 1 hook 0x55d74cef62b0
  156. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf dump hook 0x55d74cef62b0
  157. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perfcounters_schema hook 0x55d74cef62b0
  158. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf histogram dump hook 0x55d74cef62b0
  159. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 2 hook 0x55d74cef62b0
  160. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf schema hook 0x55d74cef62b0
  161. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf histogram schema hook 0x55d74cef62b0
  162. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf reset hook 0x55d74cef62b0
  163. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config show hook 0x55d74cef62b0
  164. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config help hook 0x55d74cef62b0
  165. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config set hook 0x55d74cef62b0
  166. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config unset hook 0x55d74cef62b0
  167. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config get hook 0x55d74cef62b0
  168. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config diff hook 0x55d74cef62b0
  169. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config diff get hook 0x55d74cef62b0
  170. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log flush hook 0x55d74cef62b0
  171. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log dump hook 0x55d74cef62b0
  172. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log reopen hook 0x55d74cef62b0
  173. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_mempools hook 0x55d74cf2c2c8
  174. -485> 2019-11-06 08:17:13.465 7fa9b39d4000 10 monclient: get_monmap_and_config
  175. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: build_initial_monmap
  176. -485> 2019-11-06 08:17:13.545 7fa9a356a700 2 Event(0x55d74d07e480 nevent=5000 time_id=1).set_owner idx=2 owner=140366566565632
  177. -485> 2019-11-06 08:17:13.545 7fa9a456c700 2 Event(0x55d74cf09480 nevent=5000 time_id=1).set_owner idx=0 owner=140366583351040
  178. -485> 2019-11-06 08:17:13.545 7fa9a3d6b700 2 Event(0x55d74cf08c80 nevent=5000 time_id=1).set_owner idx=1 owner=140366574958336
  179. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 1 Processor -- start
  180. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 1 -- - start start
  181. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: init
  182. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 5 adding auth protocol: cephx
  183. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: auth_supported 2 method cephx
  184. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-k8s-node-01/keyring
  185. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient: _reopen_session rank -1
  186. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-01 con 0x55d74d0c6c00 addr 192.168.200.17:6789/0
  187. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-02 con 0x55d74d0c7200 addr 192.168.200.19:6789/0
  188. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 1 -- - --> 192.168.200.17:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef0fc0 con 0
  189. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 1 -- - --> 192.168.200.19:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef1200 con 0
  190. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): _renew_subs
  191. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): authenticate will time out at 2019-11-06 08:22:13.552031
  192. -485> 2019-11-06 08:17:13.549 7fa9a356a700 1 -- 192.168.200.17:0/2492633785 learned_addr learned my addr 192.168.200.17:0/2492633785
  193. -485> 2019-11-06 08:17:13.549 7fa9a356a700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  194. -485> 2019-11-06 08:17:13.549 7fa9a3d6b700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  195. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 1 0x55d74cef1b00 mon_map magic: 0 v1
  196. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 2 0x55d74cef1d40 auth_reply(proto 2 0 (0) Success) v1
  197. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74cef1b00 con 0x55d74d0c7200
  198. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  199. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): got monmap 322, mon.k8s-node-02 is now rank 2
  200. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): dump:
  201. epoch 322
  202. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  203. last_changed 2019-10-31 14:07:12.621434
  204. created 2018-04-12 14:27:23.947601
  205. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  206. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  207. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  208.  
  209. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (751933298 0 0) 0x55d74cef1d40 con 0x55d74d0c7200
  210. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): my global_id is 1754942
  211. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74cef1200 con 0
  212. -485> 2019-11-06 08:17:13.553 7fa9a3d6b700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=327 cs=1 l=1). rx mon.1 seq 1 0x55d74d16e6c0 mon_map magic: 0 v1
  213. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.1 192.168.200.17:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  214. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  215. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): got monmap 322, mon.k8s-node-01 is now rank 1
  216. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): dump:
  217. epoch 322
  218. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  219. last_changed 2019-10-31 14:07:12.621434
  220. created 2018-04-12 14:27:23.947601
  221. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  222. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  223. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  224.  
  225. -485> 2019-11-06 08:17:13.553 7fa9a3d6b700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=327 cs=1 l=1). rx mon.1 seq 2 0x55d74d16e900 auth_reply(proto 2 0 (0) Success) v1
  226. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.1 192.168.200.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2094179837 0 0) 0x55d74d16e900 con 0x55d74d0c6c00
  227. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): my global_id is 1754926
  228. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.17:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  229. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 3 0x55d74cef1200 auth_reply(proto 2 0 (0) Success) v1
  230. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (4262477806 0 0) 0x55d74cef1200 con 0x55d74d0c7200
  231. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=327 cs=1 l=1).mark_down
  232. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=327 cs=1 l=1)._stop
  233. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 monclient: found mon.k8s-node-02
  234. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _send_mon_message to mon.k8s-node-02 at 192.168.200.19:6789/0
  235. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d74d07f400 con 0
  236. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-06 08:16:43.558302)
  237. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _send_mon_message to mon.k8s-node-02 at 192.168.200.19:6789/0
  238. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55d74d16e900 con 0
  239. -485> 2019-11-06 08:17:13.553 7fa9b39d4000 5 monclient: authenticate success, global_id 1754942
  240. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 4 0x55d74d07ec00 config(0 keys) v1
  241. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 5 0x55d74d16e000 mon_map magic: 0 v1
  242. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 6 0x55d74d16e240 auth_reply(proto 2 0 (0) Success) v1
  243. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55d74d07ec00 con 0x55d74d0c7200
  244. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: handle_config config(0 keys) v1
  245. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e000 con 0x55d74d0c7200
  246. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: handle_monmap mon_map magic: 0 v1
  247. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: got monmap 322, mon.k8s-node-02 is now rank 2
  248. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: dump:
  249. epoch 322
  250. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  251. last_changed 2019-10-31 14:07:12.621434
  252. created 2018-04-12 14:27:23.947601
  253. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  254. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  255. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  256.  
  257. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1917905670 0 0) 0x55d74d16e240 con 0x55d74d0c7200
  258. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:43.563017)
  259. -485> 2019-11-06 08:17:13.557 7fa9a1566700 4 set_mon_vals no callback set
  260. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 10 monclient: get_monmap_and_config success
  261. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 10 monclient: shutdown
  262. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN pgs=2470 cs=1 l=1).mark_down
  263. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN pgs=2470 cs=1 l=1)._stop
  264. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 shutdown_connections
  265. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections mark down 192.168.200.19:6789/0 0x55d74d0c7200
  266. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections mark down 192.168.200.17:6789/0 0x55d74d0c6c00
  267. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections delete 0x55d74d0c6c00
  268. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections delete 0x55d74d0c7200
  269. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 shutdown_connections
  270. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 wait complete.
  271. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:0/2492633785 conn(0x55d74d0c6600 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
  272. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:0/2492633785 conn(0x55d74d0c6600 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
  273. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 set uid:gid to 167:167 (ceph:ceph)
  274. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mds, pid 2712
  275. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 pidfile_write: ignore empty --pid-file
  276. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) init /var/run/ceph/ceph-mds.k8s-node-01.asok
  277. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) bind_and_listen /var/run/ceph/ceph-mds.k8s-node-01.asok
  278. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 0 hook 0x55d74cef54a0
  279. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command version hook 0x55d74cef54a0
  280. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command git_version hook 0x55d74cef54a0
  281. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command help hook 0x55d74cef6510
  282. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command get_command_descriptions hook 0x55d74cef6500
  283. -485> 2019-11-06 08:17:13.657 7fa9a2568700 5 asok(0x55d74cf1e5a0) entry start
  284. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 _finish_bind bind my_inst.addr is 0.0.0.0:6801/4011954518
  285. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: build_initial_monmap
  286. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 start start
  287. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  288. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  289. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 112 MDSCacheObject
  290. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1696 CInode
  291. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *7=112
  292. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 488 inode
  293. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 504 old_inode
  294. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t
  295. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t
  296. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock *5=200
  297. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 ScatterLock *3=144
  298. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 480 CDentry
  299. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item
  300. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock
  301. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1600 CDir
  302. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *2=32
  303. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 232 fnode_t
  304. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t *2
  305. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t *2
  306. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 280 Capability
  307. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 32 xlist<>::item *2=64
  308. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 Processor -- start
  309. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: init
  310. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  311. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: auth_supported 2 method cephx
  312. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-k8s-node-01/keyring
  313. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient: _reopen_session rank -1
  314. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-02 con 0x55d74d0c6c00 addr 192.168.200.19:6789/0
  315. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-01 con 0x55d74d0c7200 addr 192.168.200.17:6789/0
  316. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef18c0 con 0
  317. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef1680 con 0
  318. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): _renew_subs
  319. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 learned_addr learned my addr 192.168.200.17:6801/4011954518
  320. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  321. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  322. -485> 2019-11-06 08:17:13.665 7fa9a356a700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  323. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 1 0x55d74d16e6c0 mon_map magic: 0 v1
  324. -485> 2019-11-06 08:17:13.665 7fa9a356a700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  325. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 2 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  326. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  327. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  328. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 1 0x55d74d16e480 mon_map magic: 0 v1
  329. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): got monmap 322, mon.k8s-node-02 is now rank 2
  330. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 2 0x55d74d16ed80 auth_reply(proto 2 0 (0) Success) v1
  331. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): dump:
  332. epoch 322
  333. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  334. last_changed 2019-10-31 14:07:12.621434
  335. created 2018-04-12 14:27:23.947601
  336. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  337. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  338. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  339.  
  340. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3407880251 0 0) 0x55d74d16eb40 con 0x55d74d0c6c00
  341. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): my global_id is 1754945
  342. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  343. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e480 con 0x55d74d0c7200
  344. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  345. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): got monmap 322, mon.k8s-node-01 is now rank 1
  346. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): dump:
  347. epoch 322
  348. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  349. last_changed 2019-10-31 14:07:12.621434
  350. created 2018-04-12 14:27:23.947601
  351. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  352. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  353. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  354.  
  355. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2441356818 0 0) 0x55d74d16ed80 con 0x55d74d0c7200
  356. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): my global_id is 1754929
  357. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16eb40 con 0
  358. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 3 0x55d74d16e6c0 auth_reply(proto 2 0 (0) Success) v1
  359. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 3 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  360. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (306644345 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  361. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x55d74d16ed80 con 0
  362. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (2631512737 0 0) 0x55d74d16eb40 con 0x55d74d0c7200
  363. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  364. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 4 0x55d74d16e6c0 auth_reply(proto 2 0 (0) Success) v1
  365. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 799+0+0 (4051321314 0 0) 0x55d74d16e6c0 con 0x55d74d0c7200
  366. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=2471 cs=1 l=1).mark_down
  367. -485> 2019-11-06 08:17:13.669 7fa9a1566700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=2471 cs=1 l=1)._stop
  368. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 monclient: found mon.k8s-node-01
  369. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  370. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d74d08aa00 con 0
  371. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 5 0x55d74d07f400 config(0 keys) v1
  372. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 6 0x55d74d16f200 mon_map magic: 0 v1
  373. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-06 08:16:43.673641)
  374. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  375. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55d74d16eb40 con 0
  376. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55d74d07f400 con 0x55d74d0c7200
  377. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 monclient: authenticate success, global_id 1754929
  378. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: wait_auth_rotating waiting (until 2019-11-06 08:17:43.673855)
  379. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: handle_config config(0 keys) v1
  380. -485> 2019-11-06 08:17:13.669 7fa99fad9700 4 set_mon_vals no callback set
  381. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16f200 con 0x55d74d0c7200
  382. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: handle_monmap mon_map magic: 0 v1
  383. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: got monmap 322, mon.k8s-node-01 is now rank 1
  384. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: dump:
  385. epoch 322
  386. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  387. last_changed 2019-10-31 14:07:12.621434
  388. created 2018-04-12 14:27:23.947601
  389. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  390. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  391. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  392.  
  393. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 7 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  394. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (666950753 0 0) 0x55d74d16eb40 con 0x55d74d0c7200
  395. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:43.674551)
  396. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: wait_auth_rotating done
  397. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: _renew_subs
  398. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  399. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({mdsmap=0+,mgrmap=0+}) v3 -- 0x55d74d169000 con 0
  400. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command status hook 0x55d74cef6430
  401. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_ops_in_flight hook 0x55d74cef6430
  402. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command ops hook 0x55d74cef6430
  403. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_blocked_ops hook 0x55d74cef6430
  404. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_historic_ops hook 0x55d74cef6430
  405. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_historic_ops_by_duration hook 0x55d74cef6430
  406. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command scrub_path hook 0x55d74cef6430
  407. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command tag path hook 0x55d74cef6430
  408. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command flush_path hook 0x55d74cef6430
  409. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command export dir hook 0x55d74cef6430
  410. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump cache hook 0x55d74cef6430
  411. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command cache status hook 0x55d74cef6430
  412. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command cache drop hook 0x55d74cef6430
  413. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump tree hook 0x55d74cef6430
  414. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump loads hook 0x55d74cef6430
  415. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump snaps hook 0x55d74cef6430
  416. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command session evict hook 0x55d74cef6430
  417. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command osdmap barrier hook 0x55d74cef6430
  418. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command session ls hook 0x55d74cef6430
  419. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command flush journal hook 0x55d74cef6430
  420. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command force_readonly hook 0x55d74cef6430
  421. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command get subtrees hook 0x55d74cef6430
  422. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag split hook 0x55d74cef6430
  423. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag merge hook 0x55d74cef6430
  424. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag ls hook 0x55d74cef6430
  425. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command openfiles ls hook 0x55d74cef6430
  426. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump inode hook 0x55d74cef6430
  427. -485> 2019-11-06 08:17:13.673 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 8 0x55d74d16f8c0 mdsmap(e 26760) v1
  428. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 8 ==== mdsmap(e 26760) v1 ==== 493+0+0 (1599422468 0 0) 0x55d74d16f8c0 con 0x55d74d0c7200
  429. -485> 2019-11-06 08:17:13.673 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 9 0x55d74d187080 mgrmap(e 3928) v1
  430. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 1
  431. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  432. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:boot seq 1 v0) v7 -- 0x55d74d0c5180 con 0
  433. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  434. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26760 from mon.1
  435. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  436. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  437. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.-1 state ???
  438. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  439. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 not in map yet
  440. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 9 ==== mgrmap(e 3928) v1 ==== 1318+0+0 (728182834 0 0) 0x55d74d187080 con 0x55d74d0c7200
  441. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc handle_mgr_map Got map version 3928
  442. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc handle_mgr_map Active mgr is now 192.168.200.17:6800/91
  443. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  444. -485> 2019-11-06 08:17:13.673 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  445. -485> 2019-11-06 08:17:13.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  446. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a300 con 0
  447. -485> 2019-11-06 08:17:13.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=292 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1ef00 mgrconfigure(period=5, threshold=5) v2
  448. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1ef00 con 0x55d74d0c8400
  449. -485> 2019-11-06 08:17:13.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  450. -485> 2019-11-06 08:17:13.677 7fa9a1566700 4 mgrc handle_mgr_configure updated stats threshold: 5
  451. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a900 con 0
  452. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).read_bulk peer close file descriptor 26
  453. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).read_until read failed
  454. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).process read tag failed
  455. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).fault on lossy channel, failing
  456. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1)._stop
  457. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  458. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  459. -485> 2019-11-06 08:17:13.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=292 cs=1 l=1).mark_down
  460. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:14.677953
  461. -485> 2019-11-06 08:17:14.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  462. -485> 2019-11-06 08:17:14.673 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a300 con 0
  463. -485> 2019-11-06 08:17:14.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  464. -485> 2019-11-06 08:17:14.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  465. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=293 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f0e0 mgrconfigure(period=5, threshold=5) v2
  466. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f0e0 con 0x55d74d0c8400
  467. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  468. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2ac00 con 0
  469. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).read_bulk peer close file descriptor 26
  470. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).read_until read failed
  471. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).process read tag failed
  472. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).fault on lossy channel, failing
  473. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1)._stop
  474. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  475. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  476. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=293 cs=1 l=1).mark_down
  477. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:15.678577
  478. -485> 2019-11-06 08:17:15.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  479. -485> 2019-11-06 08:17:15.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a900 con 0
  480. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  481. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  482. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=294 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f2c0 mgrconfigure(period=5, threshold=5) v2
  483. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f2c0 con 0x55d74d0c8400
  484. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  485. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a300 con 0
  486. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).read_bulk peer close file descriptor 26
  487. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).read_until read failed
  488. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).process read tag failed
  489. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).fault on lossy channel, failing
  490. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1)._stop
  491. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  492. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  493. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=294 cs=1 l=1).mark_down
  494. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:16.678787
  495. -485> 2019-11-06 08:17:16.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  496. -485> 2019-11-06 08:17:16.673 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2af00 con 0
  497. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  498. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  499. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=295 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1ef00 mgrconfigure(period=5, threshold=5) v2
  500. -485> 2019-11-06 08:17:16.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1ef00 con 0x55d74d0c8400
  501. -485> 2019-11-06 08:17:16.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  502. -485> 2019-11-06 08:17:16.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2ac00 con 0
  503. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).read_bulk peer close file descriptor 26
  504. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).read_until read failed
  505. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).process read tag failed
  506. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).fault on lossy channel, failing
  507. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1)._stop
  508. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  509. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  510. -485> 2019-11-06 08:17:16.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=295 cs=1 l=1).mark_down
  511. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:17.679009
  512. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 2
  513. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  514. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26760) v7 -- 0x55d74d0c5500 con 0
  515. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  516. -485> 2019-11-06 08:17:17.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  517. -485> 2019-11-06 08:17:17.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2b200 con 0
  518. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  519. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  520. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=296 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f0e0 mgrconfigure(period=5, threshold=5) v2
  521. -485> 2019-11-06 08:17:17.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f0e0 con 0x55d74d0c8400
  522. -485> 2019-11-06 08:17:17.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  523. -485> 2019-11-06 08:17:17.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a900 con 0
  524. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).read_bulk peer close file descriptor 26
  525. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).read_until read failed
  526. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).process read tag failed
  527. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).fault on lossy channel, failing
  528. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1)._stop
  529. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  530. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  531. -485> 2019-11-06 08:17:17.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=296 cs=1 l=1).mark_down
  532. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:18.679258
  533. -485> 2019-11-06 08:17:18.601 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 10 0x55d74d16fb00 mdsmap(e 26761) v1
  534. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 10 ==== mdsmap(e 26761) v1 ==== 618+0+0 (462217262 0 0) 0x55d74d16fb00 con 0x55d74d0c7200
  535. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26761 from mon.1
  536. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  537. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  538. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.0 state up:standby
  539. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  540. -485> 2019-11-06 08:17:18.605 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:boot -> up:standby
  541. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Map has assigned me to become a standby
  542. -485> 2019-11-06 08:17:18.605 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 11 0x55d74d0c5500 mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26761) v7
  543. -485> 2019-11-06 08:17:18.605 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 11 ==== mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26761) v7 ==== 140+0+0 (2252603696 0 0) 0x55d74d0c5500 con 0x55d74d0c7200
  544. -485> 2019-11-06 08:17:18.605 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:boot seq 2 rtt 0.93202
  545. -485> 2019-11-06 08:17:18.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  546. -485> 2019-11-06 08:17:18.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2b500 con 0
  547. -485> 2019-11-06 08:17:18.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  548. -485> 2019-11-06 08:17:18.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  549. -485> 2019-11-06 08:17:18.681 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=298 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f4a0 mgrconfigure(period=5, threshold=5) v2
  550. -485> 2019-11-06 08:17:18.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f4a0 con 0x55d74d0c8400
  551. -485> 2019-11-06 08:17:18.681 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  552. -485> 2019-11-06 08:17:18.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a300 con 0
  553. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 3
  554. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  555. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7 -- 0x55d74d0c5880 con 0
  556. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  557. -485> 2019-11-06 08:17:21.677 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 12 0x55d74d0c5180 mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7
  558. -485> 2019-11-06 08:17:21.677 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 12 ==== mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7 ==== 136+0+0 (1017114021 0 0) 0x55d74d0c5180 con 0x55d74d0c7200
  559. -485> 2019-11-06 08:17:21.677 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 3 rtt 0.00400009
  560. -485> 2019-11-06 08:17:23.661 7fa9a02da700 10 monclient: tick
  561. -485> 2019-11-06 08:17:23.661 7fa9a02da700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:53.664128)
  562. -485> 2019-11-06 08:17:23.681 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2b800 con 0
  563. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 4
  564. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  565. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7 -- 0x55d74d0c5c00 con 0
  566. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  567. -485> 2019-11-06 08:17:25.709 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 13 0x55d74d0c5c00 mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7
  568. -485> 2019-11-06 08:17:25.709 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 13 ==== mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7 ==== 136+0+0 (4058739488 0 0) 0x55d74d0c5c00 con 0x55d74d0c7200
  569. -485> 2019-11-06 08:17:25.709 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 4 rtt 0.0360008
  570. -485> 2019-11-06 08:17:28.441 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 14 0x55d74d16fd40 mdsmap(e 26762) v1
  571. -485> 2019-11-06 08:17:28.441 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 14 ==== mdsmap(e 26762) v1 ==== 655+0+0 (2112797438 0 0) 0x55d74d16fd40 con 0x55d74d0c7200
  572. -485> 2019-11-06 08:17:28.441 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26762 from mon.1
  573. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  574. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  575. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.0.26762 state up:replay
  576. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  577. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: initializing MDS rank 0
  578. -485> 2019-11-06 08:17:28.445 7fa9a1566700 5 asok(0x55d74cf1e5a0) register_command objecter_requests hook 0x55d74cef62a0
  579. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 monclient: _renew_subs
  580. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  581. -485> 2019-11-06 08:17:28.445 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55d74d17f000 con 0
  582. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  583. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 update_log_config log_to_monitors {default=true}
  584. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 create_logger
  585. -485> 2019-11-06 08:17:28.445 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 0
  586. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  587. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries
  588. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  589. -485> 2019-11-06 08:17:28.449 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map as rank 0
  590. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map i am now mds.0.26762
  591. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map state change up:boot --> up:replay
  592. -485> 2019-11-06 08:17:28.449 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:standby -> up:replay
  593. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 replay_start
  594. -485> 2019-11-06 08:17:28.449 7fa9a1566700 7 mds.0.cache set_recovery_set
  595. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 recovery set is
  596. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 waiting for osdmap 16073 (which blacklists prior instance)
  597. -485> 2019-11-06 08:17:28.453 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 15 0x55d74d1c6500 osd_map(16073..16073 src has 12398..16073) v4
  598. -485> 2019-11-06 08:17:28.453 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 15 ==== osd_map(16073..16073 src has 12398..16073) v4 ==== 70545+0+0 (1064777804 0 0) 0x55d74d1c6500 con 0x55d74d0c7200
  599. -485> 2019-11-06 08:17:28.457 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 16073
  600. -485> 2019-11-06 08:17:28.457 7fa9a1566700 4 mds.0.26762 handle_osd_map epoch 16073, 0 new blacklist entries
  601. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  602. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 monclient: _renew_subs
  603. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  604. -485> 2019-11-06 08:17:28.457 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({osdmap=16074}) v3 -- 0x55d74d17f000 con 0
  605. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_Wrapper
  606. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  607. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening inotable
  608. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.inotable: load
  609. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:1 6.13 6:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0840 con 0
  610. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening sessionmap
  611. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.sessionmap load
  612. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:2 6.b 6:d0630e4c:::mds0_sessionmap:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0b00 con 0
  613. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening mds log
  614. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 5 mds.0.log open discovering log bounds
  615. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening purge queue (async)
  616. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 4 mds.0.purge_queue open: opening
  617. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 mds.0.journaler.pq(ro) recover start
  618. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 mds.0.journaler.pq(ro) read_head
  619. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:3 6.55 6:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1340 con 0
  620. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: loading open file table (async)
  621. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.openfiles load
  622. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:4 6.26 6:654134d2:::mds0_openfiles.0:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1600 con 0
  623. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening snap table
  624. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.snaptable: load
  625. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:5 6.2d 6:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d18c0 con 0
  626. -485> 2019-11-06 08:17:28.461 7fa99a2ce700 10 mds.0.log _submit_thread start
  627. -485> 2019-11-06 08:17:28.461 7fa99aacf700 4 mds.0.journalpointer Reading journal pointer '400.00000000'
  628. -485> 2019-11-06 08:17:28.461 7fa9a456c700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  629. -485> 2019-11-06 08:17:28.461 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6805/12258 -- osd_op(unknown.0.26762:6 6.f 6:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1fa000 con 0
  630. -485> 2019-11-06 08:17:28.465 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6806/20250 conn(0x55d74d0c9000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  631. -485> 2019-11-06 08:17:28.469 7fa9a356a700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  632. -485> 2019-11-06 08:17:28.481 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 1 0x55d74d1d18c0 osd_op_reply(5 mds_snaptable [read 0~70] v0'0 uv68682 ondisk = 0) v8
  633. -485> 2019-11-06 08:17:28.481 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 1 ==== osd_op_reply(5 mds_snaptable [read 0~70] v0'0 uv68682 ondisk = 0) v8 ==== 157+0+70 (4058758598 0 294686788) 0x55d74d1d18c0 con 0x55d74d0c8a00
  634. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  635. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 got 70 bytes
  636. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 loaded v1
  637. -485> 2019-11-06 08:17:28.481 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 2 0x55d74d1d18c0 osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv42584 ondisk = 0) v8
  638. -485> 2019-11-06 08:17:28.481 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 2 ==== osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv42584 ondisk = 0) v8 ==== 201+0+19 (2555699017 0 1118585302) 0x55d74d1d18c0 con 0x55d74d0c8a00
  639. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: N12_GLOBAL__N_112C_IO_SM_LoadE
  640. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish loaded version 0
  641. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: omap load complete
  642. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: v 0, 0 sessions
  643. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap dump
  644. -485> 2019-11-06 08:17:28.485 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.5 seq 1 0x55d74d1fa000 osd_op_reply(6 400.00000000 [read 0~22] v0'0 uv974 ondisk = 0) v8
  645. -485> 2019-11-06 08:17:28.485 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== osd.5 192.168.200.19:6805/12258 1 ==== osd_op_reply(6 400.00000000 [read 0~22] v0'0 uv974 ondisk = 0) v8 ==== 156+0+22 (4159695775 0 3825830296) 0x55d74d1fa000 con 0x55d74d212000
  646. -485> 2019-11-06 08:17:28.485 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 3 0x55d74d1d18c0 osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv55227 ondisk = 0) v8
  647. -485> 2019-11-06 08:17:28.485 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 3 ==== osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv55227 ondisk = 0) v8 ==== 157+0+34 (1991267346 0 2626859732) 0x55d74d1d18c0 con 0x55d74d0c8a00
  648. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 mds.0.journaler.mdlog(ro) recover start
  649. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 mds.0.journaler.mdlog(ro) read_head
  650. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  651. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 got 34 bytes
  652. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 loaded v23241
  653. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:7 6.14 6:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1340 con 0
  654. -485> 2019-11-06 08:17:28.485 7fa99aacf700 4 mds.0.log Waiting for journal 0x200 to recover...
  655. -485> 2019-11-06 08:17:28.485 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 4 0x55d74d1d1340 osd_op_reply(7 200.00000000 [read 0~90] v0'0 uv1316467 ondisk = 0) v8
  656. -485> 2019-11-06 08:17:28.485 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 4 ==== osd_op_reply(7 200.00000000 [read 0~90] v0'0 uv1316467 ondisk = 0) v8 ==== 156+0+90 (132029522 0 3312475694) 0x55d74d1d1340 con 0x55d74d0c8a00
  657. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 32291946496, expire 32291946496, write 32291946496, stream_format 1). probing for end of log (from 32291946496)...
  658. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) probing for end of the log
  659. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6805/12258 -- osd_op(unknown.0.26762:8 6.32 6:4fb62f4d:::200.00001e13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1b80 con 0
  660. -485> 2019-11-06 08:17:28.489 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.5 seq 2 0x55d74d1d1b80 osd_op_reply(8 200.00001e13 [stat] v0'0 uv66449 ondisk = 0) v8
  661. -485> 2019-11-06 08:17:28.489 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== osd.5 192.168.200.19:6805/12258 2 ==== osd_op_reply(8 200.00001e13 [stat] v0'0 uv66449 ondisk = 0) v8 ==== 156+0+16 (1087123176 0 1021817632) 0x55d74d1d1b80 con 0x55d74d212000
  662. -485> 2019-11-06 08:17:28.489 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 32291946544 (header had 32291946496). recovered.
  663. -485> 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Journal 0x200 recovered.
  664. -485> 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Recovered journal 0x200 in format 1
  665. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  666. -485> 2019-11-06 08:17:28.489 7fa99aacf700 2 mds.0.26762 boot_start 1: loading/discovering base inodes
  667. -485> 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x100
  668. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x100) fetch
  669. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:9 6.33 6:cd5a64a3:::100.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1600 con 0
  670. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:10 6.3f 6:fe07bba1:::100.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1fa2c0 con 0
  671. -485> 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x1
  672. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x1) fetch
  673. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:11 6.3f 6:ff5b34d6:::1.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0840 con 0
  674. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:12 6.14 6:287034c4:::1.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0b00 con 0
  675. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 5 0x55d74d1d0b00 osd_op_reply(10 100.00000000.inode [read 0~536] v0'0 uv6657 ondisk = 0) v8
  676. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 5 ==== osd_op_reply(10 100.00000000.inode [read 0~536] v0'0 uv6657 ondisk = 0) v8 ==== 162+0+536 (1574682179 0 845223936) 0x55d74d1d0b00 con 0x55d74d0c8a00
  677. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 6 0x55d74d1d0b00 osd_op_reply(11 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
  678. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 6 ==== osd_op_reply(11 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3347466927 0 0) 0x55d74d1d0b00 con 0x55d74d0c8a00
  679. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 7 0x55d74d1d0b00 osd_op_reply(12 1.00000000.inode [read 0~482] v0'0 uv909 ondisk = 0) v8
  680. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 7 ==== osd_op_reply(12 1.00000000.inode [read 0~482] v0'0 uv909 ondisk = 0) v8 ==== 160+0+482 (3328242318 0 1134386913) 0x55d74d1d0b00 con 0x55d74d0c8a00
  681. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
  682. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482
  683. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  684. -485> 2019-11-06 08:17:28.497 7fa99b2d0700 -1 *** Caught signal (Segmentation fault) **
  685. in thread 7fa99b2d0700 thread_name:fn_anonymous
  686.  
  687. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
  688. 1: (()+0xf5f0) [0x7fa9a87eb5f0]
  689. 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x55d74bca0e52]
  690. 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x55d74bca3648]
  691. 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x55d74bc56983]
  692. 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x43) [0x55d74bc773f3]
  693. 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x16a) [0x55d74bc775aa]
  694. 7: (MDSIOContextBase::complete(int)+0x119) [0x55d74bcc3679]
  695. 8: (Finisher::finisher_thread_entry()+0x12e) [0x7fa9aabae89e]
  696. 9: (()+0x7e65) [0x7fa9a87e3e65]
  697. 10: (clone()+0x6d) [0x7fa9a78be88d]
  698. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  699.  
  700. --- logging levels ---
  701. 0/ 5 none
  702. 0/ 1 lockdep
  703. 0/ 1 context
  704. 1/ 1 crush
  705. 20/20 mds
  706. 1/ 5 mds_balancer
  707. 1/ 5 mds_locker
  708. 1/ 5 mds_log
  709. 1/ 5 mds_log_expire
  710. 1/ 5 mds_migrator
  711. 0/ 1 buffer
  712. 0/ 1 timer
  713. 0/ 1 filer
  714. 0/ 1 striper
  715. 0/ 1 objecter
  716. 0/ 5 rados
  717. 0/ 5 rbd
  718. 0/ 5 rbd_mirror
  719. 0/ 5 rbd_replay
  720. 0/ 5 journaler
  721. 0/ 5 objectcacher
  722. 0/ 5 client
  723. 1/ 5 osd
  724. 0/ 5 optracker
  725. 0/ 5 objclass
  726. 1/ 3 filestore
  727. 1/ 3 journal
  728. 0/ 5 ms
  729. 1/ 5 mon
  730. 0/10 monc
  731. 1/ 5 paxos
  732. 0/ 5 tp
  733. 1/ 5 auth
  734. 1/ 5 crypto
  735. 1/ 1 finisher
  736. 1/ 1 reserver
  737. 1/ 5 heartbeatmap
  738. 1/ 5 perfcounter
  739. 1/ 5 rgw
  740. 1/ 5 rgw_sync
  741. 1/10 civetweb
  742. 1/ 5 javaclient
  743. 1/ 5 asok
  744. 1/ 1 throttle
  745. 0/ 0 refs
  746. 1/ 5 xio
  747. 1/ 5 compressor
  748. 1/ 5 bluestore
  749. 1/ 5 bluefs
  750. 1/ 3 bdev
  751. 1/ 5 kstore
  752. 4/ 5 rocksdb
  753. 4/ 5 leveldb
  754. 4/ 5 memdb
  755. 1/ 5 kinetic
  756. 1/ 5 fuse
  757. 1/ 5 mgr
  758. 1/ 5 mgrc
  759. 1/ 5 dpdk
  760. 1/ 5 eventtrace
  761. -2/-2 (syslog threshold)
  762. 99/99 (stderr threshold)
  763. max_recent 10000
  764. max_new 1000
  765. log_file
  766. --- end dump of recent events ---
  767. --- begin dump of recent events ---
  768. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command abort hook 0x55d74cef62b0
  769. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perfcounters_dump hook 0x55d74cef62b0
  770. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 1 hook 0x55d74cef62b0
  771. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf dump hook 0x55d74cef62b0
  772. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perfcounters_schema hook 0x55d74cef62b0
  773. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf histogram dump hook 0x55d74cef62b0
  774. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 2 hook 0x55d74cef62b0
  775. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf schema hook 0x55d74cef62b0
  776. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf histogram schema hook 0x55d74cef62b0
  777. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command perf reset hook 0x55d74cef62b0
  778. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config show hook 0x55d74cef62b0
  779. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config help hook 0x55d74cef62b0
  780. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config set hook 0x55d74cef62b0
  781. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config unset hook 0x55d74cef62b0
  782. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config get hook 0x55d74cef62b0
  783. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config diff hook 0x55d74cef62b0
  784. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command config diff get hook 0x55d74cef62b0
  785. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log flush hook 0x55d74cef62b0
  786. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log dump hook 0x55d74cef62b0
  787. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command log reopen hook 0x55d74cef62b0
  788. -485> 2019-11-06 08:17:13.453 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_mempools hook 0x55d74cf2c2c8
  789. -485> 2019-11-06 08:17:13.465 7fa9b39d4000 10 monclient: get_monmap_and_config
  790. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: build_initial_monmap
  791. -485> 2019-11-06 08:17:13.545 7fa9a356a700 2 Event(0x55d74d07e480 nevent=5000 time_id=1).set_owner idx=2 owner=140366566565632
  792. -485> 2019-11-06 08:17:13.545 7fa9a456c700 2 Event(0x55d74cf09480 nevent=5000 time_id=1).set_owner idx=0 owner=140366583351040
  793. -485> 2019-11-06 08:17:13.545 7fa9a3d6b700 2 Event(0x55d74cf08c80 nevent=5000 time_id=1).set_owner idx=1 owner=140366574958336
  794. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 1 Processor -- start
  795. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 1 -- - start start
  796. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: init
  797. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 5 adding auth protocol: cephx
  798. -485> 2019-11-06 08:17:13.545 7fa9b39d4000 10 monclient: auth_supported 2 method cephx
  799. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-k8s-node-01/keyring
  800. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient: _reopen_session rank -1
  801. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-01 con 0x55d74d0c6c00 addr 192.168.200.17:6789/0
  802. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-02 con 0x55d74d0c7200 addr 192.168.200.19:6789/0
  803. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 1 -- - --> 192.168.200.17:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef0fc0 con 0
  804. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 1 -- - --> 192.168.200.19:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef1200 con 0
  805. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): _renew_subs
  806. -485> 2019-11-06 08:17:13.549 7fa9b39d4000 10 monclient(hunting): authenticate will time out at 2019-11-06 08:22:13.552031
  807. -485> 2019-11-06 08:17:13.549 7fa9a356a700 1 -- 192.168.200.17:0/2492633785 learned_addr learned my addr 192.168.200.17:0/2492633785
  808. -485> 2019-11-06 08:17:13.549 7fa9a356a700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  809. -485> 2019-11-06 08:17:13.549 7fa9a3d6b700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  810. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 1 0x55d74cef1b00 mon_map magic: 0 v1
  811. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 2 0x55d74cef1d40 auth_reply(proto 2 0 (0) Success) v1
  812. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74cef1b00 con 0x55d74d0c7200
  813. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  814. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): got monmap 322, mon.k8s-node-02 is now rank 2
  815. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): dump:
  816. epoch 322
  817. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  818. last_changed 2019-10-31 14:07:12.621434
  819. created 2018-04-12 14:27:23.947601
  820. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  821. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  822. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  823.  
  824. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (751933298 0 0) 0x55d74cef1d40 con 0x55d74d0c7200
  825. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): my global_id is 1754942
  826. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74cef1200 con 0
  827. -485> 2019-11-06 08:17:13.553 7fa9a3d6b700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=327 cs=1 l=1). rx mon.1 seq 1 0x55d74d16e6c0 mon_map magic: 0 v1
  828. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.1 192.168.200.17:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  829. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  830. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): got monmap 322, mon.k8s-node-01 is now rank 1
  831. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): dump:
  832. epoch 322
  833. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  834. last_changed 2019-10-31 14:07:12.621434
  835. created 2018-04-12 14:27:23.947601
  836. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  837. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  838. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  839.  
  840. -485> 2019-11-06 08:17:13.553 7fa9a3d6b700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=327 cs=1 l=1). rx mon.1 seq 2 0x55d74d16e900 auth_reply(proto 2 0 (0) Success) v1
  841. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.1 192.168.200.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2094179837 0 0) 0x55d74d16e900 con 0x55d74d0c6c00
  842. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient(hunting): my global_id is 1754926
  843. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.17:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  844. -485> 2019-11-06 08:17:13.553 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 3 0x55d74cef1200 auth_reply(proto 2 0 (0) Success) v1
  845. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (4262477806 0 0) 0x55d74cef1200 con 0x55d74d0c7200
  846. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=327 cs=1 l=1).mark_down
  847. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=327 cs=1 l=1)._stop
  848. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 monclient: found mon.k8s-node-02
  849. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _send_mon_message to mon.k8s-node-02 at 192.168.200.19:6789/0
  850. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d74d07f400 con 0
  851. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-06 08:16:43.558302)
  852. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 10 monclient: _send_mon_message to mon.k8s-node-02 at 192.168.200.19:6789/0
  853. -485> 2019-11-06 08:17:13.553 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 --> 192.168.200.19:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55d74d16e900 con 0
  854. -485> 2019-11-06 08:17:13.553 7fa9b39d4000 5 monclient: authenticate success, global_id 1754942
  855. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 4 0x55d74d07ec00 config(0 keys) v1
  856. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 5 0x55d74d16e000 mon_map magic: 0 v1
  857. -485> 2019-11-06 08:17:13.557 7fa9a356a700 5 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2470 cs=1 l=1). rx mon.2 seq 6 0x55d74d16e240 auth_reply(proto 2 0 (0) Success) v1
  858. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55d74d07ec00 con 0x55d74d0c7200
  859. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: handle_config config(0 keys) v1
  860. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e000 con 0x55d74d0c7200
  861. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: handle_monmap mon_map magic: 0 v1
  862. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: got monmap 322, mon.k8s-node-02 is now rank 2
  863. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: dump:
  864. epoch 322
  865. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  866. last_changed 2019-10-31 14:07:12.621434
  867. created 2018-04-12 14:27:23.947601
  868. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  869. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  870. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  871.  
  872. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 1 -- 192.168.200.17:0/2492633785 <== mon.2 192.168.200.19:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1917905670 0 0) 0x55d74d16e240 con 0x55d74d0c7200
  873. -485> 2019-11-06 08:17:13.557 7fa9a2d69700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:43.563017)
  874. -485> 2019-11-06 08:17:13.557 7fa9a1566700 4 set_mon_vals no callback set
  875. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 10 monclient: get_monmap_and_config success
  876. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 10 monclient: shutdown
  877. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN pgs=2470 cs=1 l=1).mark_down
  878. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.19:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN pgs=2470 cs=1 l=1)._stop
  879. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 shutdown_connections
  880. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections mark down 192.168.200.19:6789/0 0x55d74d0c7200
  881. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections mark down 192.168.200.17:6789/0 0x55d74d0c6c00
  882. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections delete 0x55d74d0c6c00
  883. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 5 -- 192.168.200.17:0/2492633785 shutdown_connections delete 0x55d74d0c7200
  884. -485> 2019-11-06 08:17:13.561 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 shutdown_connections
  885. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 wait complete.
  886. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 1 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:0/2492633785 conn(0x55d74d0c6600 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
  887. -485> 2019-11-06 08:17:13.569 7fa9b39d4000 2 -- 192.168.200.17:0/2492633785 >> 192.168.200.17:0/2492633785 conn(0x55d74d0c6600 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
  888. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 set uid:gid to 167:167 (ceph:ceph)
  889. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mds, pid 2712
  890. -485> 2019-11-06 08:17:13.577 7fa9b39d4000 0 pidfile_write: ignore empty --pid-file
  891. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) init /var/run/ceph/ceph-mds.k8s-node-01.asok
  892. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) bind_and_listen /var/run/ceph/ceph-mds.k8s-node-01.asok
  893. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command 0 hook 0x55d74cef54a0
  894. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command version hook 0x55d74cef54a0
  895. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command git_version hook 0x55d74cef54a0
  896. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command help hook 0x55d74cef6510
  897. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command get_command_descriptions hook 0x55d74cef6500
  898. -485> 2019-11-06 08:17:13.657 7fa9a2568700 5 asok(0x55d74cf1e5a0) entry start
  899. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 _finish_bind bind my_inst.addr is 0.0.0.0:6801/4011954518
  900. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: build_initial_monmap
  901. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 start start
  902. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  903. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  904. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 112 MDSCacheObject
  905. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1696 CInode
  906. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *7=112
  907. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 488 inode
  908. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 504 old_inode
  909. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t
  910. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t
  911. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock *5=200
  912. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 ScatterLock *3=144
  913. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 480 CDentry
  914. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item
  915. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 SimpleLock
  916. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 1600 CDir
  917. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 16 elist<>::item *2=32
  918. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 232 fnode_t
  919. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 48 nest_info_t *2
  920. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 40 frag_info_t *2
  921. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 280 Capability
  922. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 mds.k8s-node-01 32 xlist<>::item *2=64
  923. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 1 Processor -- start
  924. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: init
  925. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 5 adding auth protocol: cephx
  926. -485> 2019-11-06 08:17:13.657 7fa9b39d4000 10 monclient: auth_supported 2 method cephx
  927. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-k8s-node-01/keyring
  928. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient: _reopen_session rank -1
  929. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-02 con 0x55d74d0c6c00 addr 192.168.200.19:6789/0
  930. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): picked mon.k8s-node-01 con 0x55d74d0c7200 addr 192.168.200.17:6789/0
  931. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef18c0 con 0
  932. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 1 -- 0.0.0.0:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 0 36 bytes epoch 0) v1 -- 0x55d74cef1680 con 0
  933. -485> 2019-11-06 08:17:13.661 7fa9b39d4000 10 monclient(hunting): _renew_subs
  934. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 learned_addr learned my addr 192.168.200.17:6801/4011954518
  935. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  936. -485> 2019-11-06 08:17:13.661 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  937. -485> 2019-11-06 08:17:13.665 7fa9a356a700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mon
  938. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 1 0x55d74d16e6c0 mon_map magic: 0 v1
  939. -485> 2019-11-06 08:17:13.665 7fa9a356a700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  940. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 2 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  941. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  942. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  943. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 1 0x55d74d16e480 mon_map magic: 0 v1
  944. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): got monmap 322, mon.k8s-node-02 is now rank 2
  945. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 2 0x55d74d16ed80 auth_reply(proto 2 0 (0) Success) v1
  946. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): dump:
  947. epoch 322
  948. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  949. last_changed 2019-10-31 14:07:12.621434
  950. created 2018-04-12 14:27:23.947601
  951. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  952. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  953. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  954.  
  955. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3407880251 0 0) 0x55d74d16eb40 con 0x55d74d0c6c00
  956. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): my global_id is 1754945
  957. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  958. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 1 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16e480 con 0x55d74d0c7200
  959. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  960. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): got monmap 322, mon.k8s-node-01 is now rank 1
  961. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): dump:
  962. epoch 322
  963. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  964. last_changed 2019-10-31 14:07:12.621434
  965. created 2018-04-12 14:27:23.947601
  966. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  967. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  968. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  969.  
  970. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2441356818 0 0) 0x55d74d16ed80 con 0x55d74d0c7200
  971. -485> 2019-11-06 08:17:13.665 7fa9a1566700 10 monclient(hunting): my global_id is 1754929
  972. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55d74d16eb40 con 0
  973. -485> 2019-11-06 08:17:13.665 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2471 cs=1 l=1). rx mon.2 seq 3 0x55d74d16e6c0 auth_reply(proto 2 0 (0) Success) v1
  974. -485> 2019-11-06 08:17:13.665 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 3 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  975. -485> 2019-11-06 08:17:13.665 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.2 192.168.200.19:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (306644345 0 0) 0x55d74d16e6c0 con 0x55d74d0c6c00
  976. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x55d74d16ed80 con 0
  977. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (2631512737 0 0) 0x55d74d16eb40 con 0x55d74d0c7200
  978. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x55d74d16e6c0 con 0
  979. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 4 0x55d74d16e6c0 auth_reply(proto 2 0 (0) Success) v1
  980. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 799+0+0 (4051321314 0 0) 0x55d74d16e6c0 con 0x55d74d0c7200
  981. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=2471 cs=1 l=1).mark_down
  982. -485> 2019-11-06 08:17:13.669 7fa9a1566700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6789/0 conn(0x55d74d0c6c00 :-1 s=STATE_OPEN pgs=2471 cs=1 l=1)._stop
  983. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 monclient: found mon.k8s-node-01
  984. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  985. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d74d08aa00 con 0
  986. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 5 0x55d74d07f400 config(0 keys) v1
  987. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 6 0x55d74d16f200 mon_map magic: 0 v1
  988. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-06 08:16:43.673641)
  989. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  990. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55d74d16eb40 con 0
  991. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55d74d07f400 con 0x55d74d0c7200
  992. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 monclient: authenticate success, global_id 1754929
  993. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: wait_auth_rotating waiting (until 2019-11-06 08:17:43.673855)
  994. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: handle_config config(0 keys) v1
  995. -485> 2019-11-06 08:17:13.669 7fa99fad9700 4 set_mon_vals no callback set
  996. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 6 ==== mon_map magic: 0 v1 ==== 451+0+0 (2605070266 0 0) 0x55d74d16f200 con 0x55d74d0c7200
  997. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: handle_monmap mon_map magic: 0 v1
  998. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: got monmap 322, mon.k8s-node-01 is now rank 1
  999. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: dump:
  1000. epoch 322
  1001. fsid 571d4bfe-2c5d-45ca-8da1-91dcaf69942c
  1002. last_changed 2019-10-31 14:07:12.621434
  1003. created 2018-04-12 14:27:23.947601
  1004. 0: 192.168.200.15:6789/0 mon.k8s-node-00
  1005. 1: 192.168.200.17:6789/0 mon.k8s-node-01
  1006. 2: 192.168.200.19:6789/0 mon.k8s-node-02
  1007.  
  1008. -485> 2019-11-06 08:17:13.669 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 7 0x55d74d16eb40 auth_reply(proto 2 0 (0) Success) v1
  1009. -485> 2019-11-06 08:17:13.669 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (666950753 0 0) 0x55d74d16eb40 con 0x55d74d0c7200
  1010. -485> 2019-11-06 08:17:13.669 7fa9a1566700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:43.674551)
  1011. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: wait_auth_rotating done
  1012. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: _renew_subs
  1013. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1014. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({mdsmap=0+,mgrmap=0+}) v3 -- 0x55d74d169000 con 0
  1015. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command status hook 0x55d74cef6430
  1016. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_ops_in_flight hook 0x55d74cef6430
  1017. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command ops hook 0x55d74cef6430
  1018. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_blocked_ops hook 0x55d74cef6430
  1019. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_historic_ops hook 0x55d74cef6430
  1020. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump_historic_ops_by_duration hook 0x55d74cef6430
  1021. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command scrub_path hook 0x55d74cef6430
  1022. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command tag path hook 0x55d74cef6430
  1023. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command flush_path hook 0x55d74cef6430
  1024. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command export dir hook 0x55d74cef6430
  1025. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump cache hook 0x55d74cef6430
  1026. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command cache status hook 0x55d74cef6430
  1027. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command cache drop hook 0x55d74cef6430
  1028. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump tree hook 0x55d74cef6430
  1029. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump loads hook 0x55d74cef6430
  1030. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump snaps hook 0x55d74cef6430
  1031. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command session evict hook 0x55d74cef6430
  1032. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command osdmap barrier hook 0x55d74cef6430
  1033. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command session ls hook 0x55d74cef6430
  1034. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command flush journal hook 0x55d74cef6430
  1035. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command force_readonly hook 0x55d74cef6430
  1036. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command get subtrees hook 0x55d74cef6430
  1037. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag split hook 0x55d74cef6430
  1038. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag merge hook 0x55d74cef6430
  1039. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dirfrag ls hook 0x55d74cef6430
  1040. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command openfiles ls hook 0x55d74cef6430
  1041. -485> 2019-11-06 08:17:13.669 7fa9b39d4000 5 asok(0x55d74cf1e5a0) register_command dump inode hook 0x55d74cef6430
  1042. -485> 2019-11-06 08:17:13.673 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 8 0x55d74d16f8c0 mdsmap(e 26760) v1
  1043. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 8 ==== mdsmap(e 26760) v1 ==== 493+0+0 (1599422468 0 0) 0x55d74d16f8c0 con 0x55d74d0c7200
  1044. -485> 2019-11-06 08:17:13.673 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 9 0x55d74d187080 mgrmap(e 3928) v1
  1045. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 1
  1046. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1047. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:boot seq 1 v0) v7 -- 0x55d74d0c5180 con 0
  1048. -485> 2019-11-06 08:17:13.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  1049. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26760 from mon.1
  1050. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1051. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1052. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.-1 state ???
  1053. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  1054. -485> 2019-11-06 08:17:13.673 7fa9a1566700 10 mds.k8s-node-01 not in map yet
  1055. -485> 2019-11-06 08:17:13.673 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 9 ==== mgrmap(e 3928) v1 ==== 1318+0+0 (728182834 0 0) 0x55d74d187080 con 0x55d74d0c7200
  1056. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc handle_mgr_map Got map version 3928
  1057. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc handle_mgr_map Active mgr is now 192.168.200.17:6800/91
  1058. -485> 2019-11-06 08:17:13.673 7fa9a1566700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1059. -485> 2019-11-06 08:17:13.673 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1060. -485> 2019-11-06 08:17:13.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1061. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a300 con 0
  1062. -485> 2019-11-06 08:17:13.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=292 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1ef00 mgrconfigure(period=5, threshold=5) v2
  1063. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1ef00 con 0x55d74d0c8400
  1064. -485> 2019-11-06 08:17:13.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1065. -485> 2019-11-06 08:17:13.677 7fa9a1566700 4 mgrc handle_mgr_configure updated stats threshold: 5
  1066. -485> 2019-11-06 08:17:13.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a900 con 0
  1067. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).read_bulk peer close file descriptor 26
  1068. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).read_until read failed
  1069. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).process read tag failed
  1070. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1).fault on lossy channel, failing
  1071. -485> 2019-11-06 08:17:13.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=292 cs=1 l=1)._stop
  1072. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  1073. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  1074. -485> 2019-11-06 08:17:13.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=292 cs=1 l=1).mark_down
  1075. -485> 2019-11-06 08:17:13.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:14.677953
  1076. -485> 2019-11-06 08:17:14.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1077. -485> 2019-11-06 08:17:14.673 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a300 con 0
  1078. -485> 2019-11-06 08:17:14.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1079. -485> 2019-11-06 08:17:14.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1080. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=293 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f0e0 mgrconfigure(period=5, threshold=5) v2
  1081. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f0e0 con 0x55d74d0c8400
  1082. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1083. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2ac00 con 0
  1084. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).read_bulk peer close file descriptor 26
  1085. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).read_until read failed
  1086. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).process read tag failed
  1087. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1).fault on lossy channel, failing
  1088. -485> 2019-11-06 08:17:14.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=293 cs=1 l=1)._stop
  1089. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  1090. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  1091. -485> 2019-11-06 08:17:14.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=293 cs=1 l=1).mark_down
  1092. -485> 2019-11-06 08:17:14.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:15.678577
  1093. -485> 2019-11-06 08:17:15.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1094. -485> 2019-11-06 08:17:15.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2a900 con 0
  1095. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1096. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1097. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=294 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f2c0 mgrconfigure(period=5, threshold=5) v2
  1098. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f2c0 con 0x55d74d0c8400
  1099. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1100. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a300 con 0
  1101. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).read_bulk peer close file descriptor 26
  1102. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).read_until read failed
  1103. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).process read tag failed
  1104. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1).fault on lossy channel, failing
  1105. -485> 2019-11-06 08:17:15.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=294 cs=1 l=1)._stop
  1106. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  1107. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  1108. -485> 2019-11-06 08:17:15.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=294 cs=1 l=1).mark_down
  1109. -485> 2019-11-06 08:17:15.677 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:16.678787
  1110. -485> 2019-11-06 08:17:16.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1111. -485> 2019-11-06 08:17:16.673 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2af00 con 0
  1112. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1113. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1114. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=295 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1ef00 mgrconfigure(period=5, threshold=5) v2
  1115. -485> 2019-11-06 08:17:16.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1ef00 con 0x55d74d0c8400
  1116. -485> 2019-11-06 08:17:16.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1117. -485> 2019-11-06 08:17:16.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2ac00 con 0
  1118. -485> 2019-11-06 08:17:16.677 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).read_bulk peer close file descriptor 26
  1119. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).read_until read failed
  1120. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).process read tag failed
  1121. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1).fault on lossy channel, failing
  1122. -485> 2019-11-06 08:17:16.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=295 cs=1 l=1)._stop
  1123. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  1124. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  1125. -485> 2019-11-06 08:17:16.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=295 cs=1 l=1).mark_down
  1126. -485> 2019-11-06 08:17:16.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:17.679009
  1127. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:boot seq 2
  1128. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1129. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26760) v7 -- 0x55d74d0c5500 con 0
  1130. -485> 2019-11-06 08:17:17.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  1131. -485> 2019-11-06 08:17:17.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1132. -485> 2019-11-06 08:17:17.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2b200 con 0
  1133. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1134. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1135. -485> 2019-11-06 08:17:17.677 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=296 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f0e0 mgrconfigure(period=5, threshold=5) v2
  1136. -485> 2019-11-06 08:17:17.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f0e0 con 0x55d74d0c8400
  1137. -485> 2019-11-06 08:17:17.677 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1138. -485> 2019-11-06 08:17:17.677 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a900 con 0
  1139. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).read_bulk peer close file descriptor 26
  1140. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).read_until read failed
  1141. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).process read tag failed
  1142. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1).fault on lossy channel, failing
  1143. -485> 2019-11-06 08:17:17.681 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN pgs=296 cs=1 l=1)._stop
  1144. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc ms_handle_reset ms_handle_reset con 0x55d74d0c8400
  1145. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc reconnect Terminating session with 192.168.200.17:6800/91
  1146. -485> 2019-11-06 08:17:17.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CLOSED pgs=296 cs=1 l=1).mark_down
  1147. -485> 2019-11-06 08:17:17.681 7fa9a1566700 4 mgrc reconnect waiting to retry connect until 2019-11-06 08:17:18.679258
  1148. -485> 2019-11-06 08:17:18.601 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 10 0x55d74d16fb00 mdsmap(e 26761) v1
  1149. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 10 ==== mdsmap(e 26761) v1 ==== 618+0+0 (462217262 0 0) 0x55d74d16fb00 con 0x55d74d0c7200
  1150. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26761 from mon.1
  1151. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1152. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1153. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.-1.0 state up:standby
  1154. -485> 2019-11-06 08:17:18.605 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map in rankless mode
  1155. -485> 2019-11-06 08:17:18.605 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:boot -> up:standby
  1156. -485> 2019-11-06 08:17:18.605 7fa9a1566700 1 mds.k8s-node-01 Map has assigned me to become a standby
  1157. -485> 2019-11-06 08:17:18.605 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 11 0x55d74d0c5500 mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26761) v7
  1158. -485> 2019-11-06 08:17:18.605 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 11 ==== mdsbeacon(1754929/k8s-node-01 up:boot seq 2 v26761) v7 ==== 140+0+0 (2252603696 0 0) 0x55d74d0c5500 con 0x55d74d0c7200
  1159. -485> 2019-11-06 08:17:18.605 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:boot seq 2 rtt 0.93202
  1160. -485> 2019-11-06 08:17:18.673 7fa99f2d8700 4 mgrc reconnect Starting new session with 192.168.200.17:6800/91
  1161. -485> 2019-11-06 08:17:18.677 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgropen(unknown.k8s-node-01) v3 -- 0x55d74cf2b500 con 0
  1162. -485> 2019-11-06 08:17:18.677 7fa9a3d6b700 10 mds.k8s-node-01 MDSDaemon::ms_get_authorizer type=mgr
  1163. -485> 2019-11-06 08:17:18.677 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1164. -485> 2019-11-06 08:17:18.681 7fa9a3d6b700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6800/91 conn(0x55d74d0c8400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=298 cs=1 l=1). rx mgr.1754125 seq 1 0x55d74cf1f4a0 mgrconfigure(period=5, threshold=5) v2
  1165. -485> 2019-11-06 08:17:18.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mgr.1754125 192.168.200.17:6800/91 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x55d74cf1f4a0 con 0x55d74d0c8400
  1166. -485> 2019-11-06 08:17:18.681 7fa9a1566700 4 mgrc handle_mgr_configure stats_period=5
  1167. -485> 2019-11-06 08:17:18.681 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2a300 con 0
  1168. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 3
  1169. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1170. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7 -- 0x55d74d0c5880 con 0
  1171. -485> 2019-11-06 08:17:21.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  1172. -485> 2019-11-06 08:17:21.677 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 12 0x55d74d0c5180 mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7
  1173. -485> 2019-11-06 08:17:21.677 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 12 ==== mdsbeacon(1754929/k8s-node-01 up:standby seq 3 v26761) v7 ==== 136+0+0 (1017114021 0 0) 0x55d74d0c5180 con 0x55d74d0c7200
  1174. -485> 2019-11-06 08:17:21.677 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 3 rtt 0.00400009
  1175. -485> 2019-11-06 08:17:23.661 7fa9a02da700 10 monclient: tick
  1176. -485> 2019-11-06 08:17:23.661 7fa9a02da700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-11-06 08:16:53.664128)
  1177. -485> 2019-11-06 08:17:23.681 7fa99f2d8700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6800/91 -- mgrreport(unknown.k8s-node-01 +0-0 packed 6) v6 -- 0x55d74cf2b800 con 0
  1178. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 5 mds.beacon.k8s-node-01 Sending beacon up:standby seq 4
  1179. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1180. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7 -- 0x55d74d0c5c00 con 0
  1181. -485> 2019-11-06 08:17:25.673 7fa99e2d6700 20 mds.beacon.k8s-node-01 sender thread waiting interval 4s
  1182. -485> 2019-11-06 08:17:25.709 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 13 0x55d74d0c5c00 mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7
  1183. -485> 2019-11-06 08:17:25.709 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 13 ==== mdsbeacon(1754929/k8s-node-01 up:standby seq 4 v26761) v7 ==== 136+0+0 (4058739488 0 0) 0x55d74d0c5c00 con 0x55d74d0c7200
  1184. -485> 2019-11-06 08:17:25.709 7fa9a356a700 5 mds.beacon.k8s-node-01 received beacon reply up:standby seq 4 rtt 0.0360008
  1185. -485> 2019-11-06 08:17:28.441 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 14 0x55d74d16fd40 mdsmap(e 26762) v1
  1186. -485> 2019-11-06 08:17:28.441 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 14 ==== mdsmap(e 26762) v1 ==== 655+0+0 (2112797438 0 0) 0x55d74d16fd40 con 0x55d74d0c7200
  1187. -485> 2019-11-06 08:17:28.441 7fa9a1566700 1 mds.k8s-node-01 Updating MDS map to version 26762 from mon.1
  1188. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1189. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  1190. -485> 2019-11-06 08:17:28.441 7fa9a1566700 10 mds.k8s-node-01 map says I am 192.168.200.17:6801/4011954518 mds.0.26762 state up:replay
  1191. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  1192. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: initializing MDS rank 0
  1193. -485> 2019-11-06 08:17:28.445 7fa9a1566700 5 asok(0x55d74cf1e5a0) register_command objecter_requests hook 0x55d74cef62a0
  1194. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 monclient: _renew_subs
  1195. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1196. -485> 2019-11-06 08:17:28.445 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55d74d17f000 con 0
  1197. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  1198. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 update_log_config log_to_monitors {default=true}
  1199. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.0 create_logger
  1200. -485> 2019-11-06 08:17:28.445 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 0
  1201. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.purge_queue operator(): data pool 2 not found in OSDMap
  1202. -485> 2019-11-06 08:17:28.445 7fa9a1566700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries
  1203. -485> 2019-11-06 08:17:28.445 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  1204. -485> 2019-11-06 08:17:28.449 7fa9a1566700 10 mds.k8s-node-01 handle_mds_map: handling map as rank 0
  1205. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map i am now mds.0.26762
  1206. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 handle_mds_map state change up:boot --> up:replay
  1207. -485> 2019-11-06 08:17:28.449 7fa9a1566700 5 mds.beacon.k8s-node-01 set_want_state: up:standby -> up:replay
  1208. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 replay_start
  1209. -485> 2019-11-06 08:17:28.449 7fa9a1566700 7 mds.0.cache set_recovery_set
  1210. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 recovery set is
  1211. -485> 2019-11-06 08:17:28.449 7fa9a1566700 1 mds.0.26762 waiting for osdmap 16073 (which blacklists prior instance)
  1212. -485> 2019-11-06 08:17:28.453 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6789/0 conn(0x55d74d0c7200 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=328 cs=1 l=1). rx mon.1 seq 15 0x55d74d1c6500 osd_map(16073..16073 src has 12398..16073) v4
  1213. -485> 2019-11-06 08:17:28.453 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 <== mon.1 192.168.200.17:6789/0 15 ==== osd_map(16073..16073 src has 12398..16073) v4 ==== 70545+0+0 (1064777804 0 0) 0x55d74d1c6500 con 0x55d74d0c7200
  1214. -485> 2019-11-06 08:17:28.457 7fa9a1566700 7 mds.0.server operator(): full = 0 epoch = 16073
  1215. -485> 2019-11-06 08:17:28.457 7fa9a1566700 4 mds.0.26762 handle_osd_map epoch 16073, 0 new blacklist entries
  1216. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 mds.0.server apply_blacklist: killed 0
  1217. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 monclient: _renew_subs
  1218. -485> 2019-11-06 08:17:28.457 7fa9a1566700 10 monclient: _send_mon_message to mon.k8s-node-01 at 192.168.200.17:6789/0
  1219. -485> 2019-11-06 08:17:28.457 7fa9a1566700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6789/0 -- mon_subscribe({osdmap=16074}) v3 -- 0x55d74d17f000 con 0
  1220. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_Wrapper
  1221. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  1222. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening inotable
  1223. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.inotable: load
  1224. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:1 6.13 6:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0840 con 0
  1225. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening sessionmap
  1226. -485> 2019-11-06 08:17:28.457 7fa99b2d0700 10 mds.0.sessionmap load
  1227. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:2 6.b 6:d0630e4c:::mds0_sessionmap:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0b00 con 0
  1228. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening mds log
  1229. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 5 mds.0.log open discovering log bounds
  1230. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening purge queue (async)
  1231. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 4 mds.0.purge_queue open: opening
  1232. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 mds.0.journaler.pq(ro) recover start
  1233. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 mds.0.journaler.pq(ro) read_head
  1234. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:3 6.55 6:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1340 con 0
  1235. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: loading open file table (async)
  1236. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.openfiles load
  1237. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:4 6.26 6:654134d2:::mds0_openfiles.0:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1600 con 0
  1238. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 2 mds.0.26762 boot_start 0: opening snap table
  1239. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 10 mds.0.snaptable: load
  1240. -485> 2019-11-06 08:17:28.461 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:5 6.2d 6:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d18c0 con 0
  1241. -485> 2019-11-06 08:17:28.461 7fa99a2ce700 10 mds.0.log _submit_thread start
  1242. -485> 2019-11-06 08:17:28.461 7fa99aacf700 4 mds.0.journalpointer Reading journal pointer '400.00000000'
  1243. -485> 2019-11-06 08:17:28.461 7fa9a456c700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1244. -485> 2019-11-06 08:17:28.461 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6805/12258 -- osd_op(unknown.0.26762:6 6.f 6:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1fa000 con 0
  1245. -485> 2019-11-06 08:17:28.465 7fa9a3d6b700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.17:6806/20250 conn(0x55d74d0c9000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1246. -485> 2019-11-06 08:17:28.469 7fa9a356a700 2 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  1247. -485> 2019-11-06 08:17:28.481 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 1 0x55d74d1d18c0 osd_op_reply(5 mds_snaptable [read 0~70] v0'0 uv68682 ondisk = 0) v8
  1248. -485> 2019-11-06 08:17:28.481 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 1 ==== osd_op_reply(5 mds_snaptable [read 0~70] v0'0 uv68682 ondisk = 0) v8 ==== 157+0+70 (4058758598 0 294686788) 0x55d74d1d18c0 con 0x55d74d0c8a00
  1249. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  1250. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 got 70 bytes
  1251. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.snaptable: load_2 loaded v1
  1252. -485> 2019-11-06 08:17:28.481 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 2 0x55d74d1d18c0 osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv42584 ondisk = 0) v8
  1253. -485> 2019-11-06 08:17:28.481 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 2 ==== osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv42584 ondisk = 0) v8 ==== 201+0+19 (2555699017 0 1118585302) 0x55d74d1d18c0 con 0x55d74d0c8a00
  1254. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 MDSIOContextBase::complete: N12_GLOBAL__N_112C_IO_SM_LoadE
  1255. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish loaded version 0
  1256. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: omap load complete
  1257. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap _load_finish: v 0, 0 sessions
  1258. -485> 2019-11-06 08:17:28.481 7fa99b2d0700 10 mds.0.sessionmap dump
  1259. -485> 2019-11-06 08:17:28.485 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.5 seq 1 0x55d74d1fa000 osd_op_reply(6 400.00000000 [read 0~22] v0'0 uv974 ondisk = 0) v8
  1260. -485> 2019-11-06 08:17:28.485 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== osd.5 192.168.200.19:6805/12258 1 ==== osd_op_reply(6 400.00000000 [read 0~22] v0'0 uv974 ondisk = 0) v8 ==== 156+0+22 (4159695775 0 3825830296) 0x55d74d1fa000 con 0x55d74d212000
  1261. -485> 2019-11-06 08:17:28.485 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 3 0x55d74d1d18c0 osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv55227 ondisk = 0) v8
  1262. -485> 2019-11-06 08:17:28.485 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 3 ==== osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv55227 ondisk = 0) v8 ==== 157+0+34 (1991267346 0 2626859732) 0x55d74d1d18c0 con 0x55d74d0c8a00
  1263. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 mds.0.journaler.mdlog(ro) recover start
  1264. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 mds.0.journaler.mdlog(ro) read_head
  1265. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  1266. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 got 34 bytes
  1267. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 10 mds.0.inotable: load_2 loaded v23241
  1268. -485> 2019-11-06 08:17:28.485 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:7 6.14 6:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1340 con 0
  1269. -485> 2019-11-06 08:17:28.485 7fa99aacf700 4 mds.0.log Waiting for journal 0x200 to recover...
  1270. -485> 2019-11-06 08:17:28.485 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 4 0x55d74d1d1340 osd_op_reply(7 200.00000000 [read 0~90] v0'0 uv1316467 ondisk = 0) v8
  1271. -485> 2019-11-06 08:17:28.485 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 4 ==== osd_op_reply(7 200.00000000 [read 0~90] v0'0 uv1316467 ondisk = 0) v8 ==== 156+0+90 (132029522 0 3312475694) 0x55d74d1d1340 con 0x55d74d0c8a00
  1272. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 32291946496, expire 32291946496, write 32291946496, stream_format 1). probing for end of log (from 32291946496)...
  1273. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) probing for end of the log
  1274. -485> 2019-11-06 08:17:28.485 7fa99b2d0700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.19:6805/12258 -- osd_op(unknown.0.26762:8 6.32 6:4fb62f4d:::200.00001e13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1b80 con 0
  1275. -485> 2019-11-06 08:17:28.489 7fa9a356a700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.19:6805/12258 conn(0x55d74d212000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.5 seq 2 0x55d74d1d1b80 osd_op_reply(8 200.00001e13 [stat] v0'0 uv66449 ondisk = 0) v8
  1276. -485> 2019-11-06 08:17:28.489 7fa9a356a700 1 -- 192.168.200.17:6801/4011954518 <== osd.5 192.168.200.19:6805/12258 2 ==== osd_op_reply(8 200.00001e13 [stat] v0'0 uv66449 ondisk = 0) v8 ==== 156+0+16 (1087123176 0 1021817632) 0x55d74d1d1b80 con 0x55d74d212000
  1277. -485> 2019-11-06 08:17:28.489 7fa99b2d0700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 32291946544 (header had 32291946496). recovered.
  1278. -485> 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Journal 0x200 recovered.
  1279. -485> 2019-11-06 08:17:28.489 7fa99aacf700 4 mds.0.log Recovered journal 0x200 in format 1
  1280. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  1281. -485> 2019-11-06 08:17:28.489 7fa99aacf700 2 mds.0.26762 boot_start 1: loading/discovering base inodes
  1282. -485> 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x100
  1283. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x100) fetch
  1284. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.17:6806/20250 -- osd_op(unknown.0.26762:9 6.33 6:cd5a64a3:::100.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d1600 con 0
  1285. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:10 6.3f 6:fe07bba1:::100.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1fa2c0 con 0
  1286. -485> 2019-11-06 08:17:28.489 7fa99aacf700 0 mds.0.cache creating system inode with ino:0x1
  1287. -485> 2019-11-06 08:17:28.489 7fa99aacf700 10 mds.0.cache.ino(0x1) fetch
  1288. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:11 6.3f 6:ff5b34d6:::1.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0840 con 0
  1289. -485> 2019-11-06 08:17:28.489 7fa99aacf700 1 -- 192.168.200.17:6801/4011954518 --> 192.168.200.15:6805/24523 -- osd_op(unknown.0.26762:12 6.14 6:287034c4:::1.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e16073) v8 -- 0x55d74d1d0b00 con 0
  1290. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 5 0x55d74d1d0b00 osd_op_reply(10 100.00000000.inode [read 0~536] v0'0 uv6657 ondisk = 0) v8
  1291. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 5 ==== osd_op_reply(10 100.00000000.inode [read 0~536] v0'0 uv6657 ondisk = 0) v8 ==== 162+0+536 (1574682179 0 845223936) 0x55d74d1d0b00 con 0x55d74d0c8a00
  1292. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 6 0x55d74d1d0b00 osd_op_reply(11 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
  1293. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 6 ==== osd_op_reply(11 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3347466927 0 0) 0x55d74d1d0b00 con 0x55d74d0c8a00
  1294. -485> 2019-11-06 08:17:28.493 7fa9a456c700 5 -- 192.168.200.17:6801/4011954518 >> 192.168.200.15:6805/24523 conn(0x55d74d0c8a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=1). rx osd.1 seq 7 0x55d74d1d0b00 osd_op_reply(12 1.00000000.inode [read 0~482] v0'0 uv909 ondisk = 0) v8
  1295. -485> 2019-11-06 08:17:28.493 7fa9a456c700 1 -- 192.168.200.17:6801/4011954518 <== osd.1 192.168.200.15:6805/24523 7 ==== osd_op_reply(12 1.00000000.inode [read 0~482] v0'0 uv909 ondisk = 0) v8 ==== 160+0+482 (3328242318 0 1134386913) 0x55d74d1d0b00 con 0x55d74d0c8a00
  1296. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
  1297. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482
  1298. -485> 2019-11-06 08:17:28.493 7fa99b2d0700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  1299. -485> 2019-11-06 08:17:28.497 7fa99b2d0700 -1 *** Caught signal (Segmentation fault) **
  1300. in thread 7fa99b2d0700 thread_name:fn_anonymous
  1301.  
  1302. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
  1303. 1: (()+0xf5f0) [0x7fa9a87eb5f0]
  1304. 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x55d74bca0e52]
  1305. 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x55d74bca3648]
  1306. 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x55d74bc56983]
  1307. 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x43) [0x55d74bc773f3]
  1308. 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x16a) [0x55d74bc775aa]
  1309. 7: (MDSIOContextBase::complete(int)+0x119) [0x55d74bcc3679]
  1310. 8: (Finisher::finisher_thread_entry()+0x12e) [0x7fa9aabae89e]
  1311. 9: (()+0x7e65) [0x7fa9a87e3e65]
  1312. 10: (clone()+0x6d) [0x7fa9a78be88d]
  1313. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  1314.  
  1315. --- logging levels ---
  1316. 0/ 5 none
  1317. 0/ 1 lockdep
  1318. 0/ 1 context
  1319. 1/ 1 crush
  1320. 20/20 mds
  1321. 1/ 5 mds_balancer
  1322. 1/ 5 mds_locker
  1323. 1/ 5 mds_log
  1324. 1/ 5 mds_log_expire
  1325. 1/ 5 mds_migrator
  1326. 0/ 1 buffer
  1327. 0/ 1 timer
  1328. 0/ 1 filer
  1329. 0/ 1 striper
  1330. 0/ 1 objecter
  1331. 0/ 5 rados
  1332. 0/ 5 rbd
  1333. 0/ 5 rbd_mirror
  1334. 0/ 5 rbd_replay
  1335. 0/ 5 journaler
  1336. 0/ 5 objectcacher
  1337. 0/ 5 client
  1338. 1/ 5 osd
  1339. 0/ 5 optracker
  1340. 0/ 5 objclass
  1341. 1/ 3 filestore
  1342. 1/ 3 journal
  1343. 0/ 5 ms
  1344. 1/ 5 mon
  1345. 0/10 monc
  1346. 1/ 5 paxos
  1347. 0/ 5 tp
  1348. 1/ 5 auth
  1349. 1/ 5 crypto
  1350. 1/ 1 finisher
  1351. 1/ 1 reserver
  1352. 1/ 5 heartbeatmap
  1353. 1/ 5 perfcounter
  1354. 1/ 5 rgw
  1355. 1/ 5 rgw_sync
  1356. 1/10 civetweb
  1357. 1/ 5 javaclient
  1358. 1/ 5 asok
  1359. 1/ 1 throttle
  1360. 0/ 0 refs
  1361. 1/ 5 xio
  1362. 1/ 5 compressor
  1363. 1/ 5 bluestore
  1364. 1/ 5 bluefs
  1365. 1/ 3 bdev
  1366. 1/ 5 kstore
  1367. 4/ 5 rocksdb
  1368. 4/ 5 leveldb
  1369. 4/ 5 memdb
  1370. 1/ 5 kinetic
  1371. 1/ 5 fuse
  1372. 1/ 5 mgr
  1373. 1/ 5 mgrc
  1374. 1/ 5 dpdk
  1375. 1/ 5 eventtrace
  1376. -2/-2 (syslog threshold)
  1377. 99/99 (stderr threshold)
  1378. max_recent 10000
  1379. max_new 1000
  1380. log_file /var/lib/ceph/crash/2019-11-06_08:17:28.499984Z_6e1e5ee3-33be-4cdd-abd2-f5b3af9b78f4/log
  1381. --- end dump of recent events ---
  1382. start_mds.sh: line 4: 2712 Segmentation fault /usr/bin/ceph-mds "${DAEMON_OPTS[@]}" -i "${MDS_NAME}"
  1383. true
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!