API tools faq
paste
Advertisement
viniciusferrao

RHV Hosted Engine Trashed with XFS corruption

viniciusferrao
Nov 30th, 2020
341
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 13.99 KB | None | 0 0
raw download clone embed print report
  1. # hosted-engine --console
  2. The engine VM is running on this host
  3. setlocale: No such file or directory
  4. Escape character is ^]
  5.  
  6.  
  7. Login incorrect
  8.  
  9. Give root password for maintenance
  10. (or press Control-D to continue): ******************************
  11.  
  12. [root@rhv ~]# xfs_repair -L /dev/dm-4
  13. xfs_repair -L /dev/dm-4
  14. Phase 1 - find and verify superblock...
  15. bad primary superblock - bad magic number !!!
  16.  
  17. attempting to find secondary superblock...
  18. .found candidate secondary superblock...
  19. verified secondary superblock...
  20. writing modified primary superblock
  21. sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
  22. resetting superblock realtime bitmap ino pointer to 129
  23. sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
  24. resetting superblock realtime summary ino pointer to 130
  25. Phase 2 - using internal log
  26. - zero log...
  27. ALERT: The filesystem has valuable metadata changes in a log which is being
  28. destroyed because the -L option was used.
  29. - scan filesystem freespace and inode maps...
  30. bad magic number
  31. bad on-disk superblock 3 - bad magic number
  32. primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
  33. zeroing unused portion of secondary superblock (AG #3)
  34. reset bad sb for ag 3
  35. Metadata CRC error detected at 0x55d67c8c51f4, xfs_agfl block 0x3/0x200
  36. agfl has bad CRC for ag 0
  37. Metadata CRC error detected at 0x55d67c8c51f4, xfs_agfl block 0x1e00003/0x200
  38. agfl has bad CRC for ag 3
  39. bad agbno 218105344 in agfl, agno 3
  40. bad agbno 34150400 in agfl, agno 3
  41. bad agbno 2896232448 in agfl, agno 3
  42. bad agbno 67170412 in agfl, agno 0
  43. bad agbno 2147484672 in agfl, agno 0
  44. bad agbno 34150400 in agfl, agno 0
  45. bad agbno 1439432704 in agfl, agno 0
  46. agi unlinked bucket 0 is 804672 in ag 3 (inode=51136320)
  47. agi unlinked bucket 2 is 804674 in ag 3 (inode=51136322)
  48. agi unlinked bucket 14 is 804686 in ag 3 (inode=51136334)
  49. agi unlinked bucket 3 is 29507 in ag 2 (inode=33583939)
  50. agi unlinked bucket 10 is 6410 in ag 1 (inode=16783626)
  51. sb_icount 0, counted 7488
  52. sb_ifree 0, counted 1780
  53. sb_fdblocks 5240296, counted 4992357
  54. - found root inode chunk
  55. Phase 3 - for each AG...
  56. - scan and clear agi unlinked lists...
  57. - process known inodes and perform inode discovery...
  58. - agno = 0
  59. - agno = 1
  60. - agno = 2
  61. - agno = 3
  62. data fork in ino 50331809 claims free block 6296246
  63. data fork in ino 50331809 claims free block 6296247
  64. data fork in ino 50331811 claims free block 6296277
  65. data fork in ino 50331811 claims free block 6296278
  66. imap claims a free inode 50332786 is in use, correcting imap and clearing inode
  67. cleared inode 50332786
  68. imap claims a free inode 50332791 is in use, correcting imap and clearing inode
  69. cleared inode 50332791
  70. imap claims a free inode 50332792 is in use, correcting imap and clearing inode
  71. cleared inode 50332792
  72. data fork in ino 50334382 claims free block 6312168
  73. - process newly discovered inodes...
  74. Phase 4 - check for duplicate blocks...
  75. - setting up duplicate extent list...
  76. - check for inodes claiming duplicate blocks...
  77. - agno = 0
  78. - agno = 1
  79. - agno = 2
  80. - agno = 3
  81. entry "global.stat" in shortform directory 50332100 references free inode 50332786
  82. junking entry "global.stat" in directory inode 50332100
  83. entry "db_0.stat" in shortform directory 50332100 references free inode 50332791
  84. junking entry "db_0.stat" in directory inode 50332100
  85. entry "db_16385.stat" in shortform directory 50332100 references free inode 50332792
  86. junking entry "db_16385.stat" in directory inode 50332100
  87. Phase 5 - rebuild AG headers and trees...
  88. - reset superblock...
  89. Phase 6 - check inode connectivity...
  90. - resetting contents of realtime bitmap and summary inodes
  91. - traversing filesystem ...
  92. - traversal finished ...
  93. - moving disconnected inodes to lost+found ...
  94. disconnected inode 16783626, moving to lost+found
  95. disconnected inode 33583939, moving to lost+found
  96. disconnected inode 51136320, moving to lost+found
  97. disconnected inode 51136322, moving to lost+found
  98. disconnected inode 51136334, moving to lost+found
  99. Phase 7 - verify and correct link counts...
  100. Note - stripe unit (0) and width (0) were copied from a backup superblock.
  101. Please reset with mount -o sunit=<value>,swidth=<value> if necessary
  102. Maximum metadata LSN (478:9329) is ahead of log (1:2).
  103. Format log to cycle 481.
  104. done
  105. [root@rhv ~]# xfs_repair -L /dev/dm-3
  106. xfs_repair -L /dev/dm-3
  107. Phase 1 - find and verify superblock...
  108. Phase 2 - using internal log
  109. - zero log...
  110. ALERT: The filesystem has valuable metadata changes in a log which is being
  111. destroyed because the -L option was used.
  112. - scan filesystem freespace and inode maps...
  113. bad magic number
  114. bad on-disk superblock 3 - bad magic number
  115. primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
  116. zeroing unused portion of secondary superblock (AG #3)
  117. reset bad sb for ag 3
  118. Metadata CRC error detected at 0x55d23e59f1f4, xfs_agfl block 0xf00003/0x200
  119. agfl has bad CRC for ag 3
  120. bad agbno 1918988402 in agfl, agno 3
  121. bad agbno 1869243747 in agfl, agno 3
  122. bad agbno 1949200242 in agfl, agno 3
  123. bad agbno 1731164009 in agfl, agno 3
  124. agi unlinked bucket 51 is 12211 in ag 3 (inode=25178035)
  125. agi unlinked bucket 58 is 12218 in ag 3 (inode=25178042)
  126. agi unlinked bucket 59 is 12219 in ag 3 (inode=25178043)
  127. sb_icount 576, counted 704
  128. sb_ifree 131, counted 149
  129. sb_fdblocks 2557231, counted 2564218
  130. - found root inode chunk
  131. Phase 3 - for each AG...
  132. - scan and clear agi unlinked lists...
  133. - process known inodes and perform inode discovery...
  134. - agno = 0
  135. - agno = 1
  136. - agno = 2
  137. - agno = 3
  138. - process newly discovered inodes...
  139. Phase 4 - check for duplicate blocks...
  140. - setting up duplicate extent list...
  141. - check for inodes claiming duplicate blocks...
  142. - agno = 1
  143. - agno = 0
  144. - agno = 3
  145. - agno = 2
  146. Phase 5 - rebuild AG headers and trees...
  147. - reset superblock...
  148. Phase 6 - check inode connectivity...
  149. - resetting contents of realtime bitmap and summary inodes
  150. - traversing filesystem ...
  151. - traversal finished ...
  152. - moving disconnected inodes to lost+found ...
  153. disconnected inode 25178035, moving to lost+found
  154. disconnected inode 25178042, moving to lost+found
  155. disconnected inode 25178043, moving to lost+found
  156. Phase 7 - verify and correct link counts...
  157. Maximum metadata LSN (56:469) is ahead of log (1:2).
  158. Format log to cycle 59.
  159. done
  160. [root@rhv ~]# mount -a
  161. mount -a
  162. [ 326.170941] XFS (dm-4): Mounting V5 Filesystem
  163. [ 326.404788] XFS (dm-4): Ending clean mount
  164. [ 326.415291] XFS (dm-3): Mounting V5 Filesystem
  165. [ 326.611673] XFS (dm-3): Ending clean mount
  166. [ 326.621705] XFS (dm-2): Mounting V5 Filesystem
  167. [ 326.784067] XFS (dm-2): Starting recovery (logdev: internal)
  168. [ 326.792083] XFS (dm-2): Metadata CRC error detected at xfs_agi_read_verify+0xc7/0xf0 [xfs], xfs_agi block 0x2
  169. [ 326.794445] XFS (dm-2): Unmount and run xfs_repair
  170. [ 326.795557] XFS (dm-2): First 128 bytes of corrupted metadata buffer:
  171. [ 326.797055] 00000000: 4d 33 44 34 39 56 00 00 80 00 00 00 f0 cf 00 00 M3D49V..........
  172. [ 326.799685] 00000010: 00 00 00 00 02 00 00 00 23 10 00 00 3d 08 01 08 ........#...=...
  173. [ 326.802290] 00000020: 21 27 44 34 39 56 00 00 00 d0 00 00 01 00 00 00 !'D49V..........
  174. [ 326.804748] 00000030: 50 00 00 00 00 00 00 00 23 10 00 00 41 01 08 08 P.......#...A...
  175. [ 326.807296] 00000040: 21 27 44 34 39 56 00 00 10 d0 00 00 02 00 00 00 !'D49V..........
  176. [ 326.809883] 00000050: 60 00 00 00 00 00 00 00 23 10 00 00 41 01 08 08 `.......#...A...
  177. [ 326.812345] 00000060: 61 2f 44 34 39 56 00 00 00 00 00 00 00 00 00 00 a/D49V..........
  178. [ 326.814831] 00000070: 50 34 00 00 00 00 00 00 23 10 00 00 82 08 08 04 P4......#.......
  179. [ 326.817237] XFS (dm-2): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x2 len 1 error 74
  180. mount: /var/log/audit: mount(2) system call failed: Structure needs cleaning.
  181. [root@rhv ~]# xfs_repair -L /dev/dm-2
  182. xfs_repair -L /dev/dm-2
  183. Phase 1 - find and verify superblock...
  184. Phase 2 - using internal log
  185. - zero log...
  186. - scan filesystem freespace and inode maps...
  187. Metadata CRC error detected at 0x55cffb48abf2, xfs_agi block 0x2/0x200
  188. bad magic # 0x4d334434 for agi 0
  189. bad version # 961937408 for agi 0
  190. bad sequence # -2147483648 for agi 0
  191. bad length # -254869504 for agi 0, should be 65536
  192. bad uuid 60d00000-7034-0000-6034-000000000000 for agi 0
  193. reset bad agi for ag 0
  194. Metadata CRC error detected at 0x55cffb45e1f4, xfs_agfl block 0x3/0x200
  195. agfl has bad CRC for ag 0
  196. bad agbno 961937408 in agfl, agno 0
  197. bad agbno 3234856960 in agfl, agno 0
  198. bad agbno 3503292416 in agfl, agno 0
  199. bad agbno 2419326976 in agfl, agno 0
  200. bad agbno 33554432 for inobt root, agno 0
  201. bad agbno 2147483648 for finobt root, agno 0
  202. agi_freecount 1023934728, counted 0 in ag 0
  203. agi_freecount 1023934728, counted 0 in ag 0 finobt
  204. agi unlinked bucket 0 is 13631488 in ag 0 (inode=13631488)
  205. agi unlinked bucket 1 is 16777216 in ag 0 (inode=16777216)
  206. agi unlinked bucket 2 is 1342177280 in ag 0 (inode=1342177280)
  207. agi unlinked bucket 3 is 0 in ag 0 (inode=0)
  208. agi unlinked bucket 4 is 588251136 in ag 0 (inode=588251136)
  209. agi unlinked bucket 5 is 1090586632 in ag 0 (inode=1090586632)
  210. agi unlinked bucket 6 is 556221492 in ag 0 (inode=556221492)
  211. agi unlinked bucket 7 is 961937408 in ag 0 (inode=961937408)
  212. agi unlinked bucket 8 is 282066944 in ag 0 (inode=282066944)
  213. agi unlinked bucket 9 is 33554432 in ag 0 (inode=33554432)
  214. agi unlinked bucket 10 is 1610612736 in ag 0 (inode=1610612736)
  215. agi unlinked bucket 11 is 0 in ag 0 (inode=0)
  216. agi unlinked bucket 12 is 588251136 in ag 0 (inode=588251136)
  217. agi unlinked bucket 13 is 1090586632 in ag 0 (inode=1090586632)
  218. agi unlinked bucket 14 is 1630487604 in ag 0 (inode=1630487604)
  219. agi unlinked bucket 15 is 961937408 in ag 0 (inode=961937408)
  220. agi unlinked bucket 16 is 0 in ag 0 (inode=0)
  221. agi unlinked bucket 17 is 0 in ag 0 (inode=0)
  222. agi unlinked bucket 18 is 1345585152 in ag 0 (inode=1345585152)
  223. agi unlinked bucket 19 is 0 in ag 0 (inode=0)
  224. agi unlinked bucket 20 is 588251136 in ag 0 (inode=588251136)
  225. agi unlinked bucket 21 is 2181564420 in ag 0 (inode=2181564420)
  226. agi unlinked bucket 22 is 2149729332 in ag 0 (inode=2149729332)
  227. agi unlinked bucket 23 is 961937408 in ag 0 (inode=961937408)
  228. agi unlinked bucket 24 is 550502400 in ag 0 (inode=550502400)
  229. agi unlinked bucket 25 is 1345585152 in ag 0 (inode=1345585152)
  230. agi unlinked bucket 26 is 1614020608 in ag 0 (inode=1614020608)
  231. agi unlinked bucket 27 is 201326592 in ag 0 (inode=201326592)
  232. agi unlinked bucket 28 is 588251136 in ag 0 (inode=588251136)
  233. agi unlinked bucket 29 is 1191248898 in ag 0 (inode=1191248898)
  234. agi unlinked bucket 30 is 1295205428 in ag 0 (inode=1295205428)
  235. agi unlinked bucket 31 is 961937408 in ag 0 (inode=961937408)
  236. agi unlinked bucket 32 is 2147483648 in ag 0 (inode=2147483648)
  237. agi unlinked bucket 33 is 818937856 in ag 0 (inode=818937856)
  238. agi unlinked bucket 34 is 0 in ag 0 (inode=0)
  239. agi unlinked bucket 35 is 33554432 in ag 0 (inode=33554432)
  240. agi unlinked bucket 36 is 605028352 in ag 0 (inode=605028352)
  241. agi unlinked bucket 37 is 1023934728 in ag 0 (inode=1023934728)
  242. agi unlinked bucket 38 is 556221492 in ag 0 (inode=556221492)
  243. agi unlinked bucket 39 is 961937408 in ag 0 (inode=961937408)
  244. agi unlinked bucket 40 is 1087373312 in ag 0 (inode=1087373312)
  245. agi unlinked bucket 41 is 16777216 in ag 0 (inode=16777216)
  246. agi unlinked bucket 42 is 1342177280 in ag 0 (inode=1342177280)
  247. agi unlinked bucket 43 is 0 in ag 0 (inode=0)
  248. agi unlinked bucket 44 is 605028352 in ag 0 (inode=605028352)
  249. agi unlinked bucket 45 is 1090586632 in ag 0 (inode=1090586632)
  250. agi unlinked bucket 46 is 556221492 in ag 0 (inode=556221492)
  251. agi unlinked bucket 47 is 961937408 in ag 0 (inode=961937408)
  252. agi unlinked bucket 48 is 1355808768 in ag 0 (inode=1355808768)
  253. agi unlinked bucket 49 is 33554432 in ag 0 (inode=33554432)
  254. agi unlinked bucket 50 is 1610612736 in ag 0 (inode=1610612736)
  255. agi unlinked bucket 51 is 0 in ag 0 (inode=0)
  256. agi unlinked bucket 52 is 605028352 in ag 0 (inode=605028352)
  257. agi unlinked bucket 53 is 1090586632 in ag 0 (inode=1090586632)
  258. agi unlinked bucket 54 is 1630487604 in ag 0 (inode=1630487604)
  259. agi unlinked bucket 55 is 961937408 in ag 0 (inode=961937408)
  260. agi unlinked bucket 56 is 0 in ag 0 (inode=0)
  261. agi unlinked bucket 57 is 0 in ag 0 (inode=0)
  262. agi unlinked bucket 58 is 1882456064 in ag 0 (inode=1882456064)
  263. agi unlinked bucket 59 is 0 in ag 0 (inode=0)
  264. agi unlinked bucket 60 is 605028352 in ag 0 (inode=605028352)
  265. agi unlinked bucket 61 is 2181564420 in ag 0 (inode=2181564420)
  266. agi unlinked bucket 62 is 2351055924 in ag 0 (inode=2351055924)
  267. agi unlinked bucket 63 is 961937408 in ag 0 (inode=961937408)
  268. sb_icount 64, counted 0
  269. sb_ifree 60, counted 0
  270. sb_fdblocks 248237, counted 258246
  271. root inode chunk not found
  272. Phase 3 - for each AG...
  273. - scan and clear agi unlinked lists...
  274. - process known inodes and perform inode discovery...
  275. - agno = 0
  276. correcting imap
  277. - agno = 1
  278. - agno = 2
  279. - agno = 3
  280. - process newly discovered inodes...
  281. Phase 4 - check for duplicate blocks...
  282. - setting up duplicate extent list...
  283. - check for inodes claiming duplicate blocks...
  284. - agno = 0
  285. - agno = 1
  286. - agno = 2
  287. - agno = 3
  288. Phase 5 - rebuild AG headers and trees...
  289. - reset superblock...
  290. Phase 6 - check inode connectivity...
  291. - resetting contents of realtime bitmap and summary inodes
  292. - traversing filesystem ...
  293. - traversal finished ...
  294. - moving disconnected inodes to lost+found ...
  295. Phase 7 - verify and correct link counts...
  296. Maximum metadata LSN (2:8790) is ahead of log (1:2).
  297. Format log to cycle 5.
  298. done
  299. [root@rhv ~]# mount -a
  300. mount -a
  301. [ 392.765382] XFS (dm-2): Mounting V5 Filesystem
  302. [ 393.005352] XFS (dm-2): Ending clean mount
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!