API tools faq
paste
Advertisement
arthurd2

MDS Crash with debug 10

arthurd2
Sep 22nd, 2019
249
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 68.50 KB | None | 0 0
raw download clone embed print report
  1. cat /var/lib/ceph/crash/2019-09-23_03:42:59.443857Z_7adf164a-1d5c-4a58-bdf7-073fee7084f7/log
  2. --- begin dump of recent events ---
  3. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command abort hook 0x16140c0
  4. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perfcounters_dump hook 0x16140c0
  5. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command 1 hook 0x16140c0
  6. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perf dump hook 0x16140c0
  7. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perfcounters_schema hook 0x16140c0
  8. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perf histogram dump hook 0x16140c0
  9. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command 2 hook 0x16140c0
  10. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perf schema hook 0x16140c0
  11. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perf histogram schema hook 0x16140c0
  12. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command perf reset hook 0x16140c0
  13. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config show hook 0x16140c0
  14. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config help hook 0x16140c0
  15. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config set hook 0x16140c0
  16. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config unset hook 0x16140c0
  17. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config get hook 0x16140c0
  18. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config diff hook 0x16140c0
  19. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command config diff get hook 0x16140c0
  20. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command log flush hook 0x16140c0
  21. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command log dump hook 0x16140c0
  22. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command log reopen hook 0x16140c0
  23. -447> 2019-09-23 00:42:54.039 7f83a22b1000 5 asok(0x1674900) register_command dump_mempools hook 0x16e2068
  24. -447> 2019-09-23 00:42:54.043 7f83a22b1000 10 monclient: get_monmap_and_config
  25. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: build_initial_monmap
  26. -447> 2019-09-23 00:42:54.047 7f83948d0700 2 Event(0x16742c0 nevent=5000 time_id=1).set_owner idx=0 owner=140203109713664
  27. -447> 2019-09-23 00:42:54.047 7f83940cf700 2 Event(0x1674500 nevent=5000 time_id=1).set_owner idx=1 owner=140203101320960
  28. -447> 2019-09-23 00:42:54.047 7f83938ce700 2 Event(0x1674e00 nevent=5000 time_id=1).set_owner idx=2 owner=140203092928256
  29. -447> 2019-09-23 00:42:54.047 7f83a22b1000 1 Processor -- start
  30. -447> 2019-09-23 00:42:54.047 7f83a22b1000 1 -- - start start
  31. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: init
  32. -447> 2019-09-23 00:42:54.047 7f83a22b1000 5 adding auth protocol: cephx
  33. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: auth_supported 2 method cephx
  34. -447> 2019-09-23 00:42:54.047 7f83a22b1000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-mds1/keyring
  35. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: _reopen_session rank -1
  36. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient(hunting): picked mon.noname-b con 0x18da700 addr 192.168.8.242:6789/0
  37. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient(hunting): picked mon.noname-c con 0x18dae00 addr 192.168.8.243:6789/0
  38. -447> 2019-09-23 00:42:54.047 7f83a22b1000 1 -- - --> 192.168.8.242:6789/0 -- auth(proto 0 29 bytes epoch 0) v1 -- 0x184c780 con 0
  39. -447> 2019-09-23 00:42:54.047 7f83a22b1000 1 -- - --> 192.168.8.243:6789/0 -- auth(proto 0 29 bytes epoch 0) v1 -- 0x184ca00 con 0
  40. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient(hunting): _renew_subs
  41. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient(hunting): authenticate will time out at 2019-09-23 00:47:54.051338
  42. -447> 2019-09-23 00:42:54.047 7f83940cf700 1 -- 192.168.8.195:0/2525656140 learned_addr learned my addr 192.168.8.195:0/2525656140
  43. -447> 2019-09-23 00:42:54.047 7f83938ce700 2 -- 192.168.8.195:0/2525656140 >> 192.168.8.243:6789/0 conn(0x18dae00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  44. -447> 2019-09-23 00:42:54.047 7f83940cf700 2 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0
  45. -447> 2019-09-23 00:42:54.047 7f83938ce700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.243:6789/0 conn(0x18dae00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=66455297 cs=1 l=1). rx mon.2 seq 1 0x1675d40 mon_map magic: 0 v1
  46. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.2 192.168.8.243:6789/0 1 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x1675d40 con 0x18dae00
  47. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  48. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): got monmap 13, mon.noname-c is now rank -1
  49. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 1 0x1675f80 mon_map magic: 0 v1
  50. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 2 0x184c780 auth_reply(proto 2 0 (0) Success) v1
  51. -447> 2019-09-23 00:42:54.047 7f83938ce700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.243:6789/0 conn(0x18dae00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=66455297 cs=1 l=1). rx mon.2 seq 2 0x184ca00 auth_reply(proto 2 0 (0) Success) v1
  52. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): dump:
  53. epoch 13
  54. fsid b8205875-e56f-4280-9e52-6aab9c758586
  55. last_changed 2019-04-02 12:21:30.672734
  56. created 2016-07-21 15:54:35.663028
  57. 0: 192.168.8.241:6789/0 mon.mon1
  58. 1: 192.168.8.242:6789/0 mon.mon2
  59. 2: 192.168.8.243:6789/0 mon.mon3
  60.  
  61. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 1 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x1675f80 con 0x18da700
  62. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  63. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): got monmap 13, mon.mon2 is now rank 1
  64. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): dump:
  65. epoch 13
  66. fsid b8205875-e56f-4280-9e52-6aab9c758586
  67. last_changed 2019-04-02 12:21:30.672734
  68. created 2016-07-21 15:54:35.663028
  69. 0: 192.168.8.241:6789/0 mon.mon1
  70. 1: 192.168.8.242:6789/0 mon.mon2
  71. 2: 192.168.8.243:6789/0 mon.mon3
  72.  
  73. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.2 192.168.8.243:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (202247891 0 0) 0x184ca00 con 0x18dae00
  74. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): my global_id is 318813278
  75. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 --> 192.168.8.243:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x184cf00 con 0
  76. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3313700803 0 0) 0x184c780 con 0x18da700
  77. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient(hunting): my global_id is 319321840
  78. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 --> 192.168.8.242:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x184ca00 con 0
  79. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 3 0x184ca00 auth_reply(proto 2 0 (0) Success) v1
  80. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (20502405 0 0) 0x184ca00 con 0x18da700
  81. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 >> 192.168.8.243:6789/0 conn(0x18dae00 :-1 s=STATE_OPEN pgs=66455297 cs=1 l=1).mark_down
  82. -447> 2019-09-23 00:42:54.047 7f83930cd700 2 -- 192.168.8.195:0/2525656140 >> 192.168.8.243:6789/0 conn(0x18dae00 :-1 s=STATE_OPEN pgs=66455297 cs=1 l=1)._stop
  83. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 monclient: found mon.mon2
  84. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  85. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 --> 192.168.8.242:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1674fc0 con 0
  86. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-23 00:42:24.052894)
  87. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  88. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 --> 192.168.8.242:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x184c780 con 0
  89. -447> 2019-09-23 00:42:54.047 7f83a22b1000 5 monclient: authenticate success, global_id 319321840
  90. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 4 0x1674fc0 config(0 keys) v1
  91. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x1674fc0 con 0x18da700
  92. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient: handle_config config(0 keys) v1
  93. -447> 2019-09-23 00:42:54.047 7f83918ca700 4 set_mon_vals no callback set
  94. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 5 0x1675f80 mon_map magic: 0 v1
  95. -447> 2019-09-23 00:42:54.047 7f83940cf700 5 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105197 cs=1 l=1). rx mon.1 seq 6 0x184c780 auth_reply(proto 2 0 (0) Success) v1
  96. -447> 2019-09-23 00:42:54.047 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 5 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x1675f80 con 0x18da700
  97. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: get_monmap_and_config success
  98. -447> 2019-09-23 00:42:54.047 7f83a22b1000 10 monclient: shutdown
  99. -447> 2019-09-23 00:42:54.047 7f83930cd700 10 monclient: handle_monmap mon_map magic: 0 v1
  100. -447> 2019-09-23 00:42:54.051 7f83930cd700 10 monclient: got monmap 13, mon.mon2 is now rank 1
  101. -447> 2019-09-23 00:42:54.051 7f83930cd700 10 monclient: dump:
  102. epoch 13
  103. fsid b8205875-e56f-4280-9e52-6aab9c758586
  104. last_changed 2019-04-02 12:21:30.672734
  105. created 2016-07-21 15:54:35.663028
  106. 0: 192.168.8.241:6789/0 mon.mon1
  107. 1: 192.168.8.242:6789/0 mon.mon2
  108. 2: 192.168.8.243:6789/0 mon.mon3
  109.  
  110. -447> 2019-09-23 00:42:54.051 7f83930cd700 1 -- 192.168.8.195:0/2525656140 <== mon.1 192.168.8.242:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3456796455 0 0) 0x184c780 con 0x18da700
  111. -447> 2019-09-23 00:42:54.051 7f83930cd700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-09-23 00:42:24.053761)
  112. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN pgs=63105197 cs=1 l=1).mark_down
  113. -447> 2019-09-23 00:42:54.051 7f83a22b1000 2 -- 192.168.8.195:0/2525656140 >> 192.168.8.242:6789/0 conn(0x18da700 :-1 s=STATE_OPEN pgs=63105197 cs=1 l=1)._stop
  114. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/2525656140 shutdown_connections
  115. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 -- 192.168.8.195:0/2525656140 shutdown_connections mark down 192.168.8.243:6789/0 0x18dae00
  116. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 -- 192.168.8.195:0/2525656140 shutdown_connections mark down 192.168.8.242:6789/0 0x18da700
  117. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 -- 192.168.8.195:0/2525656140 shutdown_connections delete 0x18da700
  118. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 -- 192.168.8.195:0/2525656140 shutdown_connections delete 0x18dae00
  119. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/2525656140 shutdown_connections
  120. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/2525656140 wait complete.
  121. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/2525656140 >> 192.168.8.195:0/2525656140 conn(0x18da000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
  122. -447> 2019-09-23 00:42:54.051 7f83a22b1000 2 -- 192.168.8.195:0/2525656140 >> 192.168.8.195:0/2525656140 conn(0x18da000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
  123. -447> 2019-09-23 00:42:54.051 7f83a22b1000 0 set uid:gid to 64045:64045 (ceph:ceph)
  124. -447> 2019-09-23 00:42:54.051 7f83a22b1000 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mds, pid 854821
  125. -447> 2019-09-23 00:42:54.051 7f83a22b1000 0 pidfile_write: ignore empty --pid-file
  126. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) init /var/run/ceph/ceph-mds.mds1.asok
  127. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) bind_and_listen /var/run/ceph/ceph-mds.mds1.asok
  128. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) register_command 0 hook 0x1611330
  129. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) register_command version hook 0x1611330
  130. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) register_command git_version hook 0x1611330
  131. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) register_command help hook 0x16142f0
  132. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 asok(0x1674900) register_command get_command_descriptions hook 0x1614300
  133. -447> 2019-09-23 00:42:54.051 7f83928cc700 5 asok(0x1674900) entry start
  134. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:0/0 learned_addr learned my addr 192.168.8.195:0/0
  135. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:6800/1451096063 _finish_bind bind my_inst.addr is 192.168.8.195:6800/1451096063
  136. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient: build_initial_monmap
  137. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:6800/1451096063 start start
  138. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 adding auth protocol: cephx
  139. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 adding auth protocol: cephx
  140. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 112 MDSCacheObject
  141. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 1784 CInode
  142. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 16 elist<>::item *7=112
  143. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 536 inode
  144. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 552 old_inode
  145. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 48 nest_info_t
  146. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 40 frag_info_t
  147. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 40 SimpleLock *5=200
  148. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 48 ScatterLock *3=144
  149. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 512 CDentry
  150. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 16 elist<>::item
  151. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 40 SimpleLock
  152. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 1608 CDir
  153. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 16 elist<>::item *2=32
  154. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 232 fnode_t
  155. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 48 nest_info_t *2
  156. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 40 frag_info_t *2
  157. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 288 Capability
  158. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 mds.mds1 32 xlist<>::item *2=64
  159. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 Processor -- start
  160. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient: init
  161. -447> 2019-09-23 00:42:54.051 7f83a22b1000 5 adding auth protocol: cephx
  162. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient: auth_supported 2 method cephx
  163. -447> 2019-09-23 00:42:54.051 7f83a22b1000 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-mds1/keyring
  164. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient: _reopen_session rank -1
  165. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient(hunting): picked mon.noname-c con 0x18da700 addr 192.168.8.243:6789/0
  166. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient(hunting): picked mon.noname-b con 0x18db500 addr 192.168.8.242:6789/0
  167. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- auth(proto 0 29 bytes epoch 0) v1 -- 0x184cc80 con 0
  168. -447> 2019-09-23 00:42:54.051 7f83a22b1000 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.243:6789/0 -- auth(proto 0 29 bytes epoch 0) v1 -- 0x184c780 con 0
  169. -447> 2019-09-23 00:42:54.051 7f83a22b1000 10 monclient(hunting): _renew_subs
  170. -447> 2019-09-23 00:42:54.051 7f83940cf700 10 mds.mds1 MDSDaemon::ms_get_authorizer type=mon
  171. -447> 2019-09-23 00:42:54.051 7f83938ce700 10 mds.mds1 MDSDaemon::ms_get_authorizer type=mon
  172. -447> 2019-09-23 00:42:54.051 7f83938ce700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  173. -447> 2019-09-23 00:42:54.051 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  174. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 1 0x1674fc0 mon_map magic: 0 v1
  175. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 2 0x184cf00 auth_reply(proto 2 0 (0) Success) v1
  176. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 1 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x1674fc0 con 0x18db500
  177. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  178. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): got monmap 13, mon.noname-b is now rank -1
  179. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): dump:
  180. epoch 13
  181. fsid b8205875-e56f-4280-9e52-6aab9c758586
  182. last_changed 2019-04-02 12:21:30.672734
  183. created 2016-07-21 15:54:35.663028
  184. 0: 192.168.8.241:6789/0 mon.mon1
  185. 1: 192.168.8.242:6789/0 mon.mon2
  186. 2: 192.168.8.243:6789/0 mon.mon3
  187.  
  188. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1882862728 0 0) 0x184cf00 con 0x18db500
  189. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): my global_id is 319321843
  190. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x184d900 con 0
  191. -447> 2019-09-23 00:42:54.055 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=66455298 cs=1 l=1). rx mon.2 seq 1 0x16761c0 mon_map magic: 0 v1
  192. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.2 192.168.8.243:6789/0 1 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x16761c0 con 0x18da700
  193. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  194. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): got monmap 13, mon.mon3 is now rank 2
  195. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): dump:
  196. epoch 13
  197. fsid b8205875-e56f-4280-9e52-6aab9c758586
  198. last_changed 2019-04-02 12:21:30.672734
  199. created 2016-07-21 15:54:35.663028
  200. 0: 192.168.8.241:6789/0 mon.mon1
  201. 1: 192.168.8.242:6789/0 mon.mon2
  202. 2: 192.168.8.243:6789/0 mon.mon3
  203.  
  204. -447> 2019-09-23 00:42:54.055 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=66455298 cs=1 l=1). rx mon.2 seq 2 0x184c780 auth_reply(proto 2 0 (0) Success) v1
  205. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.2 192.168.8.243:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3811178035 0 0) 0x184c780 con 0x18da700
  206. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient(hunting): my global_id is 318813281
  207. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.243:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x184cf00 con 0
  208. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 3 0x184d900 auth_reply(proto 2 0 (0) Success) v1
  209. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (328361507 0 0) 0x184d900 con 0x18db500
  210. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x184c780 con 0
  211. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 4 0x184c780 auth_reply(proto 2 0 (0) Success) v1
  212. -447> 2019-09-23 00:42:54.055 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=66455298 cs=1 l=1). rx mon.2 seq 3 0x184cf00 auth_reply(proto 2 0 (0) Success) v1
  213. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (3122030673 0 0) 0x184c780 con 0x18db500
  214. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_OPEN pgs=66455298 cs=1 l=1).mark_down
  215. -447> 2019-09-23 00:42:54.055 7f83918ca700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6789/0 conn(0x18da700 :-1 s=STATE_OPEN pgs=66455298 cs=1 l=1)._stop
  216. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 monclient: found mon.mon2
  217. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  218. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1675440 con 0
  219. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-23 00:42:24.058650)
  220. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  221. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x184cf00 con 0
  222. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 monclient: authenticate success, global_id 319321843
  223. -447> 2019-09-23 00:42:54.055 7f83a22b1000 10 monclient: wait_auth_rotating waiting (until 2019-09-23 00:43:24.058683)
  224. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 5 0x1675440 config(0 keys) v1
  225. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x1675440 con 0x18db500
  226. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: handle_config config(0 keys) v1
  227. -447> 2019-09-23 00:42:54.055 7f838f717700 4 set_mon_vals no callback set
  228. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 6 0x1675f80 mon_map magic: 0 v1
  229. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 6 ==== mon_map magic: 0 v1 ==== 388+0+0 (3535606529 0 0) 0x1675f80 con 0x18db500
  230. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: handle_monmap mon_map magic: 0 v1
  231. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: got monmap 13, mon.mon2 is now rank 1
  232. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: dump:
  233. epoch 13
  234. fsid b8205875-e56f-4280-9e52-6aab9c758586
  235. last_changed 2019-04-02 12:21:30.672734
  236. created 2016-07-21 15:54:35.663028
  237. 0: 192.168.8.241:6789/0 mon.mon1
  238. 1: 192.168.8.242:6789/0 mon.mon2
  239. 2: 192.168.8.243:6789/0 mon.mon3
  240.  
  241. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 7 0x184cf00 auth_reply(proto 2 0 (0) Success) v1
  242. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (2011348902 0 0) 0x184cf00 con 0x18db500
  243. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-09-23 00:42:24.059066)
  244. -447> 2019-09-23 00:42:54.055 7f83a22b1000 10 monclient: wait_auth_rotating done
  245. -447> 2019-09-23 00:42:54.055 7f83a22b1000 10 monclient: _renew_subs
  246. -447> 2019-09-23 00:42:54.055 7f83a22b1000 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  247. -447> 2019-09-23 00:42:54.055 7f83a22b1000 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mon_subscribe({mdsmap=0+,mgrmap=0+}) v3 -- 0x1675680 con 0
  248. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command status hook 0x1614450
  249. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump_ops_in_flight hook 0x1614450
  250. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command ops hook 0x1614450
  251. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump_blocked_ops hook 0x1614450
  252. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump_historic_ops hook 0x1614450
  253. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump_historic_ops_by_duration hook 0x1614450
  254. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command scrub_path hook 0x1614450
  255. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command tag path hook 0x1614450
  256. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command flush_path hook 0x1614450
  257. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command export dir hook 0x1614450
  258. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump cache hook 0x1614450
  259. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command cache status hook 0x1614450
  260. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command cache drop hook 0x1614450
  261. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump tree hook 0x1614450
  262. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump loads hook 0x1614450
  263. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump snaps hook 0x1614450
  264. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command session evict hook 0x1614450
  265. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command osdmap barrier hook 0x1614450
  266. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command session ls hook 0x1614450
  267. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command flush journal hook 0x1614450
  268. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command force_readonly hook 0x1614450
  269. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command get subtrees hook 0x1614450
  270. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dirfrag split hook 0x1614450
  271. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dirfrag merge hook 0x1614450
  272. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dirfrag ls hook 0x1614450
  273. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command openfiles ls hook 0x1614450
  274. -447> 2019-09-23 00:42:54.055 7f83a22b1000 5 asok(0x1674900) register_command dump inode hook 0x1614450
  275. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 8 0x184cc80 mdsmap(e 280339) v1
  276. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 8 ==== mdsmap(e 280339) v1 ==== 493+0+0 (4121765133 0 0) 0x184cc80 con 0x18db500
  277. -447> 2019-09-23 00:42:54.055 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 9 0x1670600 mgrmap(e 1093) v1
  278. -447> 2019-09-23 00:42:54.055 7f838df14700 5 mds.beacon.mds1 Sending beacon up:boot seq 1
  279. -447> 2019-09-23 00:42:54.055 7f838df14700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  280. -447> 2019-09-23 00:42:54.055 7f838df14700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mdsbeacon(319321843/mds1 up:boot seq 1 v0) v7 -- 0x190e380 con 0
  281. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 mds.mds1 Updating MDS map to version 280339 from mon.1
  282. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 mds.mds1 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  283. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 mds.mds1 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  284. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 mds.mds1 map says I am 192.168.8.195:6800/1451096063 mds.-1.-1 state ???
  285. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 mds.mds1 handle_mds_map: handling map in rankless mode
  286. -447> 2019-09-23 00:42:54.055 7f83918ca700 10 mds.mds1 not in map yet
  287. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 9 ==== mgrmap(e 1093) v1 ==== 1913+0+0 (3003451980 0 0) 0x1670600 con 0x18db500
  288. -447> 2019-09-23 00:42:54.055 7f83918ca700 4 mgrc handle_mgr_map Got map version 1093
  289. -447> 2019-09-23 00:42:54.055 7f83918ca700 4 mgrc handle_mgr_map Active mgr is now 192.168.8.243:6800/3565567
  290. -447> 2019-09-23 00:42:54.055 7f83918ca700 4 mgrc reconnect Starting new session with 192.168.8.243:6800/3565567
  291. -447> 2019-09-23 00:42:54.055 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.243:6800/3565567 -- mgropen(unknown.mds1) v3 -- 0x1926000 con 0
  292. -447> 2019-09-23 00:42:54.055 7f83940cf700 10 mds.mds1 MDSDaemon::ms_get_authorizer type=mgr
  293. -447> 2019-09-23 00:42:54.055 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6800/3565567 conn(0x18dca00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  294. -447> 2019-09-23 00:42:54.059 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.243:6800/3565567 conn(0x18dca00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1196823 cs=1 l=1). rx mgr.316285979 seq 1 0x16541e0 mgrconfigure(period=5, threshold=5) v2
  295. -447> 2019-09-23 00:42:54.059 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mgr.316285979 192.168.8.243:6800/3565567 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x16541e0 con 0x18dca00
  296. -447> 2019-09-23 00:42:54.059 7f83918ca700 4 mgrc handle_mgr_configure stats_period=5
  297. -447> 2019-09-23 00:42:54.059 7f83918ca700 4 mgrc handle_mgr_configure updated stats threshold: 5
  298. -447> 2019-09-23 00:42:54.059 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.243:6800/3565567 -- mgrreport(unknown.mds1 +0-0 packed 6) v6 -- 0x190e700 con 0
  299. -447> 2019-09-23 00:42:58.059 7f838df14700 5 mds.beacon.mds1 Sending beacon up:boot seq 2
  300. -447> 2019-09-23 00:42:58.059 7f838df14700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  301. -447> 2019-09-23 00:42:58.059 7f838df14700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mdsbeacon(319321843/mds1 up:boot seq 2 v280339) v7 -- 0x190ea80 con 0
  302. -447> 2019-09-23 00:42:59.059 7f838ef16700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.243:6800/3565567 -- mgrreport(unknown.mds1 +0-0 packed 6) v6 -- 0x190f180 con 0
  303. -447> 2019-09-23 00:42:59.267 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 10 0x184d900 mdsmap(e 280340) v1
  304. -447> 2019-09-23 00:42:59.267 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 10 ==== mdsmap(e 280340) v1 ==== 611+0+0 (1416820368 0 0) 0x184d900 con 0x18db500
  305. -447> 2019-09-23 00:42:59.267 7f83918ca700 1 mds.mds1 Updating MDS map to version 280340 from mon.1
  306. -447> 2019-09-23 00:42:59.267 7f83918ca700 10 mds.mds1 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  307. -447> 2019-09-23 00:42:59.267 7f83918ca700 10 mds.mds1 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  308. -447> 2019-09-23 00:42:59.267 7f83918ca700 10 mds.mds1 map says I am 192.168.8.195:6800/1451096063 mds.-1.0 state up:standby
  309. -447> 2019-09-23 00:42:59.267 7f83918ca700 10 mds.mds1 handle_mds_map: handling map in rankless mode
  310. -447> 2019-09-23 00:42:59.267 7f83918ca700 5 mds.beacon.mds1 set_want_state: up:boot -> up:standby
  311. -447> 2019-09-23 00:42:59.267 7f83918ca700 1 mds.mds1 Map has assigned me to become a standby
  312. -447> 2019-09-23 00:42:59.267 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 11 0x190ea80 mdsbeacon(319321843/mds1 up:boot seq 2 v280340) v7
  313. -447> 2019-09-23 00:42:59.267 7f83938ce700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 11 ==== mdsbeacon(319321843/mds1 up:boot seq 2 v280340) v7 ==== 133+0+0 (1477658002 0 0) 0x190ea80 con 0x18db500
  314. -447> 2019-09-23 00:42:59.267 7f83938ce700 5 mds.beacon.mds1 received beacon reply up:boot seq 2 rtt 1.20804
  315. -447> 2019-09-23 00:42:59.431 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 12 0x184c780 mdsmap(e 280341) v1
  316. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 12 ==== mdsmap(e 280341) v1 ==== 702+0+0 (3984739174 0 0) 0x184c780 con 0x18db500
  317. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.mds1 Updating MDS map to version 280341 from mon.1
  318. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.mds1 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
  319. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.mds1 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
  320. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.mds1 map says I am 192.168.8.195:6800/1451096063 mds.0.280341 state up:replay
  321. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 77 not found in OSDMap
  322. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 79 not found in OSDMap
  323. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 87 not found in OSDMap
  324. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 94 not found in OSDMap
  325. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 99 not found in OSDMap
  326. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 100 not found in OSDMap
  327. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 101 not found in OSDMap
  328. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 102 not found in OSDMap
  329. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.mds1 handle_mds_map: initializing MDS rank 0
  330. -447> 2019-09-23 00:42:59.431 7f83918ca700 5 asok(0x1674900) register_command objecter_requests hook 0x1614560
  331. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 monclient: _renew_subs
  332. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  333. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x1676880 con 0
  334. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 log_channel(cluster) update_config to_monitors: true to_syslog: true syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
  335. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.0.0 update_log_config log_to_monitors {default=true}
  336. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.0.0 create_logger
  337. -447> 2019-09-23 00:42:59.431 7f83918ca700 7 mds.0.server operator(): full = 0 epoch = 0
  338. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 77 not found in OSDMap
  339. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 79 not found in OSDMap
  340. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 87 not found in OSDMap
  341. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 94 not found in OSDMap
  342. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 99 not found in OSDMap
  343. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 100 not found in OSDMap
  344. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 101 not found in OSDMap
  345. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.purge_queue operator(): data pool 102 not found in OSDMap
  346. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries
  347. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.0.server apply_blacklist: killed 0
  348. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.mds1 handle_mds_map: handling map as rank 0
  349. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.0.280341 handle_mds_map i am now mds.0.280341
  350. -447> 2019-09-23 00:42:59.431 7f83918ca700 7 mds.0.server required_client_features: 0000000000000080
  351. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.0.280341 handle_mds_map state change up:boot --> up:replay
  352. -447> 2019-09-23 00:42:59.431 7f83918ca700 5 mds.beacon.mds1 set_want_state: up:standby -> up:replay
  353. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.0.280341 replay_start
  354. -447> 2019-09-23 00:42:59.431 7f83918ca700 7 mds.0.cache set_recovery_set
  355. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.0.280341 recovery set is
  356. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 mds.0.280341 waiting for osdmap 296617 (which blacklists prior instance)
  357. -447> 2019-09-23 00:42:59.431 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.242:6789/0 conn(0x18db500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=63105198 cs=1 l=1). rx mon.1 seq 13 0x16e38c0 osd_map(296617..296617 src has 296056..296617) v4
  358. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 <== mon.1 192.168.8.242:6789/0 13 ==== osd_map(296617..296617 src has 296056..296617) v4 ==== 503134+0+0 (2058033359 0 0) 0x16e38c0 con 0x18db500
  359. -447> 2019-09-23 00:42:59.431 7f83918ca700 7 mds.0.server operator(): full = 0 epoch = 296617
  360. -447> 2019-09-23 00:42:59.431 7f83918ca700 4 mds.0.280341 handle_osd_map epoch 296617, 0 new blacklist entries
  361. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 mds.0.server apply_blacklist: killed 0
  362. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 monclient: _renew_subs
  363. -447> 2019-09-23 00:42:59.431 7f83918ca700 10 monclient: _send_mon_message to mon.mon2 at 192.168.8.242:6789/0
  364. -447> 2019-09-23 00:42:59.431 7f83918ca700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.242:6789/0 -- mon_subscribe({osdmap=296618}) v3 -- 0x1676ac0 con 0
  365. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 MDSIOContextBase::complete: 12C_IO_Wrapper
  366. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  367. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: opening inotable
  368. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 mds.0.inotable: load
  369. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.209:6821/630769 -- osd_op(unknown.0.280341:1 76.13 76:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1926680 con 0
  370. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: opening sessionmap
  371. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 mds.0.sessionmap load
  372. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.207:6810/1406990 -- osd_op(unknown.0.280341:2 76.b 76:d0630e4c:::mds0_sessionmap:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x19269c0 con 0
  373. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: opening mds log
  374. -447> 2019-09-23 00:42:59.431 7f838af0e700 5 mds.0.log open discovering log bounds
  375. -447> 2019-09-23 00:42:59.431 7f838a70d700 4 mds.0.journalpointer Reading journal pointer '400.00000000'
  376. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: opening purge queue (async)
  377. -447> 2019-09-23 00:42:59.431 7f838af0e700 4 mds.0.purge_queue open: opening
  378. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 mds.0.journaler.pq(ro) recover start
  379. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 mds.0.journaler.pq(ro) read_head
  380. -447> 2019-09-23 00:42:59.431 7f8389f0c700 10 mds.0.log _submit_thread start
  381. -447> 2019-09-23 00:42:59.431 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.201:6808/1927 -- osd_op(unknown.0.280341:3 76.f 76:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1927040 con 0
  382. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.216:6810/2642264 -- osd_op(unknown.0.280341:4 76.55 76:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1926d00 con 0
  383. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: loading open file table (async)
  384. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 mds.0.openfiles load
  385. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.215:6801/7987 -- osd_op(unknown.0.280341:5 76.26 76:654134d2:::mds0_openfiles.0:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1927380 con 0
  386. -447> 2019-09-23 00:42:59.431 7f838af0e700 2 mds.0.280341 boot_start 0: opening snap table
  387. -447> 2019-09-23 00:42:59.431 7f838af0e700 10 mds.0.snaptable: load
  388. -447> 2019-09-23 00:42:59.431 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.205:6801/652466 -- osd_op(unknown.0.280341:6 76.2d 76:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x19276c0 con 0
  389. -447> 2019-09-23 00:42:59.431 7f83948d0700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6821/630769 conn(0x18dd100 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  390. -447> 2019-09-23 00:42:59.431 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.207:6810/1406990 conn(0x18dd800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  391. -447> 2019-09-23 00:42:59.431 7f83938ce700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.205:6801/652466 conn(0x1aaf500 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  392. -447> 2019-09-23 00:42:59.435 7f83938ce700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.201:6808/1927 conn(0x1aae000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  393. -447> 2019-09-23 00:42:59.435 7f83948d0700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.216:6810/2642264 conn(0x1aae700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  394. -447> 2019-09-23 00:42:59.435 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.215:6801/7987 conn(0x1aaee00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  395. -447> 2019-09-23 00:42:59.435 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.205:6801/652466 conn(0x1aaf500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5703 cs=1 l=1). rx osd.507 seq 1 0x1927040 osd_op_reply(6 mds_snaptable [read 0~70] v0'0 uv6203167 ondisk = 0) v8
  396. -447> 2019-09-23 00:42:59.435 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6821/630769 conn(0x18dd100 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5534 cs=1 l=1). rx osd.26 seq 1 0x1926680 osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv6307278 ondisk = 0) v8
  397. -447> 2019-09-23 00:42:59.435 7f83938ce700 1 -- 192.168.8.195:6800/1451096063 <== osd.507 192.168.8.205:6801/652466 1 ==== osd_op_reply(6 mds_snaptable [read 0~70] v0'0 uv6203167 ondisk = 0) v8 ==== 157+0+70 (1741550263 0 2521607951) 0x1927040 con 0x1aaf500
  398. -447> 2019-09-23 00:42:59.435 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.26 192.168.8.209:6821/630769 1 ==== osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv6307278 ondisk = 0) v8 ==== 157+0+34 (3832737465 0 3268084999) 0x1926680 con 0x18dd100
  399. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  400. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.snaptable: load_2 got 70 bytes
  401. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.snaptable: load_2 loaded v11
  402. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
  403. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.inotable: load_2 got 34 bytes
  404. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.inotable: load_2 loaded v5610
  405. -447> 2019-09-23 00:42:59.435 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.216:6810/2642264 conn(0x1aae700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=6401 cs=1 l=1). rx osd.1603 seq 1 0x1926680 osd_op_reply(4 500.00000000 [read 0~90] v0'0 uv6674134 ondisk = 0) v8
  406. -447> 2019-09-23 00:42:59.435 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.1603 192.168.8.216:6810/2642264 1 ==== osd_op_reply(4 500.00000000 [read 0~90] v0'0 uv6674134 ondisk = 0) v8 ==== 156+0+90 (830642969 0 887937218) 0x1926680 con 0x1aae700
  407. -447> 2019-09-23 00:42:59.435 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.201:6808/1927 conn(0x1aae000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=8631 cs=1 l=1). rx osd.18 seq 1 0x1927040 osd_op_reply(3 400.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v8
  408. -447> 2019-09-23 00:42:59.435 7f83938ce700 1 -- 192.168.8.195:6800/1451096063 <== osd.18 192.168.8.201:6808/1927 1 ==== osd_op_reply(3 400.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (4275129107 0 3825830296) 0x1927040 con 0x1aae000
  409. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 213909504, expire 216456156, write 216456156, stream_format 1). probing for end of log (from 216456156)...
  410. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 mds.0.journaler.pq(ro) probing for end of the log
  411. -447> 2019-09-23 00:42:59.435 7f838a70d700 1 mds.0.journaler.mdlog(ro) recover start
  412. -447> 2019-09-23 00:42:59.435 7f838a70d700 1 mds.0.journaler.mdlog(ro) read_head
  413. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.207:6811/1406967 -- osd_op(unknown.0.280341:7 76.6b 76:d7d72459:::500.00000033:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e296617) v8 -- 0x1926000 con 0
  414. -447> 2019-09-23 00:42:59.435 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.202:6816/29554 -- osd_op(unknown.0.280341:8 76.14 76:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x19269c0 con 0
  415. -447> 2019-09-23 00:42:59.435 7f838a70d700 4 mds.0.log Waiting for journal 0x200 to recover...
  416. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.209:6804/630676 -- osd_op(unknown.0.280341:9 76.59 76:9a969940:::500.00000034:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e296617) v8 -- 0x1928080 con 0
  417. -447> 2019-09-23 00:42:59.435 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.207:6810/1406990 conn(0x18dd800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9929 cs=1 l=1). rx osd.12 seq 1 0x1927380 osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv6608366 ondisk = 0) v8
  418. -447> 2019-09-23 00:42:59.435 7f83940cf700 1 -- 192.168.8.195:6800/1451096063 <== osd.12 192.168.8.207:6810/1406990 1 ==== osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv6608366 ondisk = 0) v8 ==== 201+0+7183 (682573394 0 1785913711) 0x1927380 con 0x18dd800
  419. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 MDSIOContextBase::complete: N12_GLOBAL__N_112C_IO_SM_LoadE
  420. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap _load_finish loaded version 1884251
  421. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap _load_finish: omap load complete
  422. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap _load_finish: v 1884251, 25 sessions
  423. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap dump
  424. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318832293 0x1af4940 state open completed {} prealloc_inos [] used_inos []
  425. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318546447 0x1af41c0 state open completed {19=0x0} prealloc_inos [0x10000b3e00e~0x45,0x10000b3ff87~0x321,0x10000b441ac~0x7b] used_inos []
  426. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318546441 0x1af3e00 state open completed {} prealloc_inos [0x10000b355a2~0x12f,0x10000b384ef~0x10c,0x10000b3de6b~0x19b] used_inos []
  427. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318538578 0x1af3a40 state open completed {37651=0x0,37652=0x0} prealloc_inos [] used_inos []
  428. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318070480 0x1af32c0 state open completed {} prealloc_inos [0x10000b43b0b~0x6b,0x10000b43f9e~0x19,0x10000b446a0~0x1f5] used_inos []
  429. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.299527899 0x17fa1c0 state open completed {} prealloc_inos [] used_inos []
  430. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.300673074 0x17fa940 state open completed {} prealloc_inos [0x10000b3b40d~0x3de] used_inos []
  431. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.296613681 0x17f9e00 state open completed {} prealloc_inos [0x10000b30b3a~0x138,0x10000b352e7~0x1f5] used_inos []
  432. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.149650701 0x17f9680 state open completed {} prealloc_inos [] used_inos []
  433. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.317848570 0x1af2f00 state open completed {} prealloc_inos [0x10000b4376c~0x20,0x10000b44227~0x1f5] used_inos []
  434. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.317704545 0x1af2780 state open completed {100=0x10000b42f23} prealloc_inos [0x10000b42f24~0x3e2] used_inos []
  435. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318832254 0x1af4580 state open completed {} prealloc_inos [0x10000b2faae~0x27,0x10000b2fd76~0x135,0x10000b44611~0x8f,0x10000b44895~0x1f8] used_inos []
  436. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.317796234 0x1af2b40 state open completed {} prealloc_inos [0x10000b40d66~0x100,0x10000b41051~0xe,0x10000b41a26~0xbd,0x10000b43b76~0x1f5] used_inos []
  437. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.302501361 0x17fad00 state open completed {} prealloc_inos [] used_inos []
  438. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.310980946 0x17fb0c0 state open completed {} prealloc_inos [0x10000b3027f~0x2a,0x10000b3144f~0x1eb,0x10000b316bd~0x172,0x10000b31ba0~0x2e,0x10000b3eb84~0x1a] used_inos []
  439. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.161005875 0x17f9a40 state open completed {} prealloc_inos [0x10000a42e11~0x2a7,0x10000b1931a~0xb4,0x10000b25a54~0x15] used_inos []
  440. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.300068976 0x17fa580 state open completed {} prealloc_inos [0x10000b3b7f9~0x3db] used_inos []
  441. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.315281235 0x17fb480 state open completed {162310=0x10000b44fc6,162311=0x10000b44fc7} prealloc_inos [0x10000b44fc8~0xa4,0x10000b45261~0x1f5] used_inos []
  442. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.318197115 0x1af3680 state open completed {} prealloc_inos [0x10000b429bd~0x177,0x10000b4378c~0x1,0x10000b43f60~0x3e,0x10000b4506c~0x1f5] used_inos []
  443. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.315364794 0x17fb840 state open completed {} prealloc_inos [] used_inos []
  444. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.317430926 0x1af2000 state open completed {} prealloc_inos [0x10000b44b1c~0x35b] used_inos []
  445. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.316663179 0x17fbc00 state open completed {8=0x10000b41060} prealloc_inos [0x10000b41061~0x3e7] used_inos []
  446. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.139347171 0x17f8f00 state open completed {} prealloc_inos [] used_inos []
  447. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.141975678 0x17f92c0 state open completed {} prealloc_inos [] used_inos []
  448. -447> 2019-09-23 00:42:59.435 7f838af0e700 10 mds.0.sessionmap client.317698239 0x1af23c0 state open completed {} prealloc_inos [0x10000b42b3c~0x3e1] used_inos []
  449. -447> 2019-09-23 00:42:59.435 7f83948d0700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.207:6811/1406967 conn(0x1aafc00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  450. -447> 2019-09-23 00:42:59.435 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.207:6811/1406967 conn(0x1aafc00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10739 cs=1 l=1). rx osd.8 seq 1 0x1926000 osd_op_reply(7 500.00000033 [stat] v0'0 uv6023841 ondisk = 0) v8
  451. -447> 2019-09-23 00:42:59.435 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.8 192.168.8.207:6811/1406967 1 ==== osd_op_reply(7 500.00000033 [stat] v0'0 uv6023841 ondisk = 0) v8 ==== 156+0+16 (940143978 0 3867969935) 0x1926000 con 0x1aafc00
  452. -447> 2019-09-23 00:42:59.435 7f83938ce700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6804/630676 conn(0x1ab1100 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  453. -447> 2019-09-23 00:42:59.435 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.202:6816/29554 conn(0x1ab0300 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  454. -447> 2019-09-23 00:42:59.435 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6804/630676 conn(0x1ab1100 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=9810 cs=1 l=1). rx osd.25 seq 1 0x19276c0 osd_op_reply(9 500.00000034 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
  455. -447> 2019-09-23 00:42:59.435 7f83938ce700 1 -- 192.168.8.195:6800/1451096063 <== osd.25 192.168.8.209:6804/630676 1 ==== osd_op_reply(9 500.00000034 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (3465720754 0 0) 0x19276c0 con 0x1ab1100
  456. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 216456156 (header had 216456156). recovered.
  457. -447> 2019-09-23 00:42:59.435 7f838bf10700 4 mds.0.purge_queue operator(): open complete
  458. -447> 2019-09-23 00:42:59.435 7f838bf10700 1 mds.0.journaler.pq(ro) set_writeable
  459. -447> 2019-09-23 00:42:59.435 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.202:6816/29554 conn(0x1ab0300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4866 cs=1 l=1). rx osd.34 seq 1 0x19269c0 osd_op_reply(8 200.00000000 [read 0~90] v0'0 uv9049841 ondisk = 0) v8
  460. -447> 2019-09-23 00:42:59.435 7f83940cf700 1 -- 192.168.8.195:6800/1451096063 <== osd.34 192.168.8.202:6816/29554 1 ==== osd_op_reply(8 200.00000000 [read 0~90] v0'0 uv9049841 ondisk = 0) v8 ==== 156+0+90 (3154508031 0 1939212667) 0x19269c0 con 0x1ab0300
  461. -447> 2019-09-23 00:42:59.435 7f838af0e700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 1934891155456, expire 1934895260490, write 1935292910172, stream_format 1). probing for end of log (from 1935292910172)...
  462. -447> 2019-09-23 00:42:59.435 7f838af0e700 1 mds.0.journaler.mdlog(ro) probing for end of the log
  463. -447> 2019-09-23 00:42:59.435 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.214:6812/4153756 -- osd_op(unknown.0.280341:10 76.12 76:48a08311:::200.00070a61:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e296617) v8 -- 0x1927a00 con 0
  464. -447> 2019-09-23 00:42:59.435 7f838af0e700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.209:6803/630745 -- osd_op(unknown.0.280341:11 76.40 76:021b4c73:::200.00070a62:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e296617) v8 -- 0x1928700 con 0
  465. -447> 2019-09-23 00:42:59.435 7f83948d0700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.214:6812/4153756 conn(0x1af6000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  466. -447> 2019-09-23 00:42:59.435 7f83940cf700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6803/630745 conn(0x1af6700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  467. -447> 2019-09-23 00:42:59.435 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.214:6812/4153756 conn(0x1af6000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10241 cs=1 l=1). rx osd.1401 seq 1 0x1927380 osd_op_reply(10 200.00070a61 [stat] v0'0 uv5992703 ondisk = 0) v8
  468. -447> 2019-09-23 00:42:59.435 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.1401 192.168.8.214:6812/4153756 1 ==== osd_op_reply(10 200.00070a61 [stat] v0'0 uv5992703 ondisk = 0) v8 ==== 156+0+16 (815171025 0 2004390911) 0x1927380 con 0x1af6000
  469. -447> 2019-09-23 00:42:59.439 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6803/630745 conn(0x1af6700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10716 cs=1 l=1). rx osd.24 seq 1 0x1926680 osd_op_reply(11 200.00070a62 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
  470. -447> 2019-09-23 00:42:59.439 7f83940cf700 1 -- 192.168.8.195:6800/1451096063 <== osd.24 192.168.8.209:6803/630745 1 ==== osd_op_reply(11 200.00070a62 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (2135259478 0 0) 0x1926680 con 0x1af6700
  471. -447> 2019-09-23 00:42:59.439 7f838af0e700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1935292930823 (header had 1935292910172). recovered.
  472. -447> 2019-09-23 00:42:59.439 7f838a70d700 4 mds.0.log Journal 0x200 recovered.
  473. -447> 2019-09-23 00:42:59.439 7f838a70d700 4 mds.0.log Recovered journal 0x200 in format 1
  474. -447> 2019-09-23 00:42:59.439 7f838a70d700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
  475. -447> 2019-09-23 00:42:59.439 7f838a70d700 2 mds.0.280341 boot_start 1: loading/discovering base inodes
  476. -447> 2019-09-23 00:42:59.439 7f838a70d700 0 mds.0.cache creating system inode with ino:0x100
  477. -447> 2019-09-23 00:42:59.439 7f838a70d700 10 mds.0.cache.ino(0x100) fetch
  478. -447> 2019-09-23 00:42:59.439 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.202:6820/29654 -- osd_op(unknown.0.280341:12 76.33 76:cd5a64a3:::100.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1927d40 con 0
  479. -447> 2019-09-23 00:42:59.439 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.209:6821/630769 -- osd_op(unknown.0.280341:13 76.7f 76:fe07bba1:::100.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1929400 con 0
  480. -447> 2019-09-23 00:42:59.439 7f838a70d700 0 mds.0.cache creating system inode with ino:0x1
  481. -447> 2019-09-23 00:42:59.439 7f838a70d700 10 mds.0.cache.ino(0x1) fetch
  482. -447> 2019-09-23 00:42:59.439 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.209:6821/630769 -- osd_op(unknown.0.280341:14 76.7f 76:ff5b34d6:::1.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1929740 con 0
  483. -447> 2019-09-23 00:42:59.439 7f838a70d700 1 -- 192.168.8.195:6800/1451096063 --> 192.168.8.202:6816/29554 -- osd_op(unknown.0.280341:15 76.14 76:287034c4:::1.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e296617) v8 -- 0x1929a80 con 0
  484. -447> 2019-09-23 00:42:59.439 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6821/630769 conn(0x18dd100 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5534 cs=1 l=1). rx osd.26 seq 2 0x1926d00 osd_op_reply(13 100.00000000.inode [read 0~544] v0'0 uv6004117 ondisk = 0) v8
  485. -447> 2019-09-23 00:42:59.439 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.26 192.168.8.209:6821/630769 2 ==== osd_op_reply(13 100.00000000.inode [read 0~544] v0'0 uv6004117 ondisk = 0) v8 ==== 162+0+544 (3715994564 0 2278151428) 0x1926d00 con 0x18dd100
  486. -447> 2019-09-23 00:42:59.439 7f83940cf700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.202:6816/29554 conn(0x1ab0300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4866 cs=1 l=1). rx osd.34 seq 2 0x1929a80 osd_op_reply(15 1.00000000.inode [read 0~482] v0'0 uv7433643 ondisk = 0) v8
  487. -447> 2019-09-23 00:42:59.439 7f83940cf700 1 -- 192.168.8.195:6800/1451096063 <== osd.34 192.168.8.202:6816/29554 2 ==== osd_op_reply(15 1.00000000.inode [read 0~482] v0'0 uv7433643 ondisk = 0) v8 ==== 160+0+482 (624093978 0 2573754889) 0x1929a80 con 0x1ab0300
  488. -447> 2019-09-23 00:42:59.439 7f83948d0700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.209:6821/630769 conn(0x18dd100 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5534 cs=1 l=1). rx osd.26 seq 3 0x1926d00 osd_op_reply(14 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
  489. -447> 2019-09-23 00:42:59.439 7f83948d0700 1 -- 192.168.8.195:6800/1451096063 <== osd.26 192.168.8.209:6821/630769 3 ==== osd_op_reply(14 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (2037075132 0 0) 0x1926d00 con 0x18dd100
  490. -447> 2019-09-23 00:42:59.439 7f838af0e700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
  491. -447> 2019-09-23 00:42:59.439 7f838af0e700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482
  492. -447> 2019-09-23 00:42:59.439 7f838af0e700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011')
  493. -447> 2019-09-23 00:42:59.439 7f83938ce700 2 -- 192.168.8.195:6800/1451096063 >> 192.168.8.202:6820/29654 conn(0x1af7c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
  494. -447> 2019-09-23 00:42:59.439 7f83938ce700 5 -- 192.168.8.195:6800/1451096063 >> 192.168.8.202:6820/29654 conn(0x1af7c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1027 cs=1 l=1). rx osd.37 seq 1 0x1927d40 osd_op_reply(12 100.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
  495. -447> 2019-09-23 00:42:59.439 7f83938ce700 1 -- 192.168.8.195:6800/1451096063 <== osd.37 192.168.8.202:6820/29654 1 ==== osd_op_reply(12 100.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 156+0+0 (4131137693 0 0) 0x1927d40 con 0x1af7c00
  496. -447> 2019-09-23 00:42:59.439 7f838af0e700 -1 *** Caught signal (Segmentation fault) **
  497. in thread 7f838af0e700 thread_name:fn_anonymous
  498.  
  499. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
  500. 1: (()+0x11390) [0x7f8398e0f390]
  501. 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]
  502. 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]
  503. 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]
  504. 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]
  505. 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]
  506. 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]
  507. 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f8399584ece]
  508. 9: (()+0x76ba) [0x7f8398e056ba]
  509. 10: (clone()+0x6d) [0x7f839862e41d]
  510. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  511.  
  512. --- logging levels ---
  513. 0/ 5 none
  514. 0/ 1 lockdep
  515. 0/ 1 context
  516. 1/ 1 crush
  517. 10/10 mds
  518. 1/ 5 mds_balancer
  519. 1/ 5 mds_locker
  520. 1/ 5 mds_log
  521. 1/ 5 mds_log_expire
  522. 1/ 5 mds_migrator
  523. 0/ 1 buffer
  524. 0/ 1 timer
  525. 0/ 1 filer
  526. 0/ 1 striper
  527. 0/ 1 objecter
  528. 0/ 5 rados
  529. 0/ 5 rbd
  530. 0/ 5 rbd_mirror
  531. 0/ 5 rbd_replay
  532. 0/ 5 journaler
  533. 0/ 5 objectcacher
  534. 0/ 5 client
  535. 1/ 5 osd
  536. 0/ 5 optracker
  537. 0/ 5 objclass
  538. 1/ 3 filestore
  539. 1/ 3 journal
  540. 0/ 5 ms
  541. 1/ 5 mon
  542. 0/10 monc
  543. 1/ 5 paxos
  544. 0/ 5 tp
  545. 1/ 5 auth
  546. 1/ 5 crypto
  547. 1/ 1 finisher
  548. 1/ 1 reserver
  549. 1/ 5 heartbeatmap
  550. 1/ 5 perfcounter
  551. 1/ 5 rgw
  552. 1/ 5 rgw_sync
  553. 1/10 civetweb
  554. 1/ 5 javaclient
  555. 1/ 5 asok
  556. 1/ 1 throttle
  557. 0/ 0 refs
  558. 1/ 5 xio
  559. 1/ 5 compressor
  560. 1/ 5 bluestore
  561. 1/ 5 bluefs
  562. 1/ 3 bdev
  563. 1/ 5 kstore
  564. 4/ 5 rocksdb
  565. 4/ 5 leveldb
  566. 4/ 5 memdb
  567. 1/ 5 kinetic
  568. 1/ 5 fuse
  569. 1/ 5 mgr
  570. 1/ 5 mgrc
  571. 1/ 5 dpdk
  572. 1/ 5 eventtrace
  573. 99/99 (syslog threshold)
  574. -1/-1 (stderr threshold)
  575. max_recent 10000
  576. max_new 1000
  577. log_file /var/lib/ceph/crash/2019-09-23_03:42:59.443857Z_7adf164a-1d5c-4a58-bdf7-073fee7084f7/log
  578. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!