API tools faq
paste
Advertisement
romainl972

ceph status

romainl972
Mar 8th, 2024
83
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 41.63 KB | None | 0 0
raw download clone embed print report
  1. cluster:
  2. id: 9b4b12fe-4dc6-11ed-9ed9-d18a342d7c2b
  3. health: HEALTH_ERR
  4. 1 scrub errors
  5. Possible data damage: 1 pg inconsistent
  6. Degraded data redundancy: 95264/2020629 objects degraded (4.715%), 11 pgs degraded, 11 pgs undersized
  7.  
  8. services:
  9. mon: 3 daemons, quorum alpha-cen,gamma-cen,beta-cen (age 2w)
  10. mgr: alpha-cen.yckelo(active, since 6w), standbys: gamma-cen.rclrlv
  11. osd: 15 osds: 14 up (since 72s), 14 in (since 6m); 11 remapped pgs
  12.  
  13. data:
  14. pools: 5 pools, 129 pgs
  15. objects: 673.54k objects, 2.5 TiB
  16. usage: 6.8 TiB used, 62 TiB / 69 TiB avail
  17. pgs: 95264/2020629 objects degraded (4.715%)
  18. 118 active+clean
  19. 10 active+undersized+degraded+remapped+backfilling
  20. 1 active+undersized+degraded+remapped+inconsistent+backfilling
  21.  
  22. io:
  23. client: 152 KiB/s rd, 872 KiB/s wr, 9 op/s rd, 44 op/s wr
  24. recovery: 260 MiB/s, 74 objects/s
  25.  
  26.  
  27. 2024-03-08T23:33:38.637491-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 95110/2020629 objects degraded (4.707%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  28. 2024-03-08T23:33:43.639709-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 94817/2020629 objects degraded (4.692%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  29. 2024-03-08T23:33:48.641632-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 94248/2020629 objects degraded (4.664%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  30. 2024-03-08T23:33:53.643535-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 93938/2020629 objects degraded (4.649%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  31. 2024-03-08T23:33:58.645822-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 93483/2020629 objects degraded (4.626%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  32. 2024-03-08T23:34:03.650714-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 93147/2020629 objects degraded (4.610%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  33. 2024-03-08T23:34:08.653144-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 92691/2020629 objects degraded (4.587%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  34. 2024-03-08T23:34:13.655315-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 92339/2020629 objects degraded (4.570%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  35. 2024-03-08T23:34:18.657293-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 91799/2020629 objects degraded (4.543%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  36. 2024-03-08T23:34:23.659392-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 91465/2020629 objects degraded (4.527%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  37. 2024-03-08T23:34:28.661941-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 90938/2020629 objects degraded (4.500%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  38. 2024-03-08T23:34:33.664058-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 90702/2020629 objects degraded (4.489%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  39. 2024-03-08T23:34:38.666035-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 90119/2020629 objects degraded (4.460%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  40. 2024-03-08T23:34:43.668032-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 89824/2020629 objects degraded (4.445%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  41. 2024-03-08T23:34:48.670722-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 89357/2020629 objects degraded (4.422%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  42. 2024-03-08T23:34:53.672732-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 88965/2020629 objects degraded (4.403%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  43. 2024-03-08T23:34:58.674675-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 88515/2020629 objects degraded (4.381%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  44. 2024-03-08T23:35:03.676825-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 88081/2020629 objects degraded (4.359%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  45. 2024-03-08T23:35:08.679239-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 87637/2020629 objects degraded (4.337%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  46. 2024-03-08T23:35:13.681275-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 87437/2020629 objects degraded (4.327%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  47. 2024-03-08T23:35:18.683390-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 86944/2020629 objects degraded (4.303%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  48. 2024-03-08T23:35:23.685365-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 86657/2020629 objects degraded (4.289%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  49. 2024-03-08T23:35:28.687476-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 86192/2020629 objects degraded (4.266%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  50. 2024-03-08T23:35:33.689538-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 85994/2020629 objects degraded (4.256%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  51. 2024-03-08T23:35:38.692041-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 85489/2020629 objects degraded (4.231%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  52. 2024-03-08T23:35:43.694622-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 85186/2020629 objects degraded (4.216%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  53. 2024-03-08T23:35:48.696623-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 84716/2020629 objects degraded (4.193%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  54. 2024-03-08T23:35:53.698646-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 84512/2020629 objects degraded (4.182%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  55. 2024-03-08T23:35:55.894850-0500 mon.alpha-cen [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)
  56. 2024-03-08T23:35:58.701139-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 84104/2020629 objects degraded (4.162%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  57. 2024-03-08T23:36:03.703245-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 83633/2020629 objects degraded (4.139%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  58. 2024-03-08T23:36:08.706423-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 83172/2020629 objects degraded (4.116%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  59. 2024-03-08T23:36:13.711167-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 82734/2020629 objects degraded (4.094%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  60. 2024-03-08T23:36:18.713497-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 82345/2020629 objects degraded (4.075%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  61. 2024-03-08T23:36:23.715614-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 82044/2020629 objects degraded (4.060%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  62. 2024-03-08T23:36:28.718011-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 81635/2020629 objects degraded (4.040%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  63. 2024-03-08T23:36:33.720287-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 81206/2020629 objects degraded (4.019%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  64. 2024-03-08T23:36:38.722389-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 80769/2020629 objects degraded (3.997%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  65. 2024-03-08T23:36:43.724408-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 80380/2020629 objects degraded (3.978%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  66. 2024-03-08T23:36:48.726400-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 79995/2020632 objects degraded (3.959%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  67. 2024-03-08T23:36:53.728724-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 79547/2020632 objects degraded (3.937%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  68. 2024-03-08T23:36:58.730884-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 79140/2020632 objects degraded (3.917%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  69. 2024-03-08T23:37:03.733012-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 78801/2020632 objects degraded (3.900%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  70. 2024-03-08T23:37:08.735313-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 78376/2020632 objects degraded (3.879%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  71. 2024-03-08T23:37:13.738019-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 77963/2020632 objects degraded (3.858%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  72. 2024-03-08T23:37:18.740060-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 77523/2020632 objects degraded (3.837%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  73. 2024-03-08T23:37:23.742039-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 77226/2020632 objects degraded (3.822%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  74. 2024-03-08T23:37:28.744192-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 76684/2020632 objects degraded (3.795%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  75. 2024-03-08T23:37:33.746163-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 76384/2020632 objects degraded (3.780%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  76. 2024-03-08T23:37:38.748289-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 75954/2020632 objects degraded (3.759%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  77. 2024-03-08T23:37:43.750377-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 75544/2020632 objects degraded (3.739%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  78. 2024-03-08T23:37:48.753029-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 75090/2020632 objects degraded (3.716%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  79. 2024-03-08T23:37:53.755200-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 74686/2020632 objects degraded (3.696%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  80. 2024-03-08T23:37:58.757349-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 74194/2020632 objects degraded (3.672%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  81. 2024-03-08T23:38:03.759444-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 73902/2020632 objects degraded (3.657%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  82. 2024-03-08T23:38:08.761593-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 73374/2020632 objects degraded (3.631%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  83. 2024-03-08T23:38:13.763679-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 73034/2020632 objects degraded (3.614%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  84. 2024-03-08T23:38:18.765677-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 72475/2020632 objects degraded (3.587%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  85. 2024-03-08T23:38:23.767735-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 72163/2020632 objects degraded (3.571%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  86. 2024-03-08T23:38:28.770178-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 71752/2020632 objects degraded (3.551%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  87. 2024-03-08T23:38:33.772303-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 71406/2020632 objects degraded (3.534%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  88. 2024-03-08T23:38:38.774871-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 70957/2020632 objects degraded (3.512%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  89. 2024-03-08T23:38:43.777076-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 70515/2020632 objects degraded (3.490%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  90. 2024-03-08T23:38:48.780075-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 70085/2020632 objects degraded (3.468%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  91. 2024-03-08T23:38:53.782281-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 69638/2020632 objects degraded (3.446%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  92. 2024-03-08T23:38:58.784455-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 69263/2020632 objects degraded (3.428%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  93. 2024-03-08T23:39:03.786415-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 68836/2020632 objects degraded (3.407%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  94. 2024-03-08T23:39:08.788379-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 68434/2020632 objects degraded (3.387%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  95. 2024-03-08T23:39:13.790564-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 68011/2020632 objects degraded (3.366%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  96. 2024-03-08T23:39:18.793445-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 67715/2020632 objects degraded (3.351%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  97. 2024-03-08T23:39:23.797248-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 67404/2020632 objects degraded (3.336%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  98. 2024-03-08T23:39:28.799482-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 67081/2020632 objects degraded (3.320%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  99. 2024-03-08T23:39:33.801752-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 66546/2020632 objects degraded (3.293%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  100. 2024-03-08T23:39:38.805188-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 66297/2020632 objects degraded (3.281%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  101. 2024-03-08T23:39:43.807283-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 65797/2020632 objects degraded (3.256%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  102. 2024-03-08T23:39:48.809519-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 65498/2020632 objects degraded (3.241%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  103. 2024-03-08T23:39:53.811779-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 64945/2020632 objects degraded (3.214%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  104. 2024-03-08T23:39:58.817027-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 64468/2020632 objects degraded (3.190%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  105. 2024-03-08T23:40:00.000202-0500 mon.alpha-cen [ERR] Health detail: HEALTH_ERR 1 failed cephadm daemon(s); 1 scrub errors; Possible data damage: 1 pg inconsistent; Degraded data redundancy: 64468/2020632 objects degraded (3.190%), 11 pgs degraded, 11 pgs undersized
  106. 2024-03-08T23:40:00.000276-0500 mon.alpha-cen [ERR] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
  107. 2024-03-08T23:40:00.000291-0500 mon.alpha-cen [ERR] daemon osd.3 on alpha-cen is in error state
  108. 2024-03-08T23:40:00.000302-0500 mon.alpha-cen [ERR] [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
  109. 2024-03-08T23:40:00.000314-0500 mon.alpha-cen [ERR] [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
  110. 2024-03-08T23:40:00.000335-0500 mon.alpha-cen [ERR] pg 2.1b is active+undersized+degraded+remapped+inconsistent+backfilling, acting [0,11]
  111. 2024-03-08T23:40:00.000353-0500 mon.alpha-cen [ERR] [WRN] PG_DEGRADED: Degraded data redundancy: 64468/2020632 objects degraded (3.190%), 11 pgs degraded, 11 pgs undersized
  112. 2024-03-08T23:40:00.000367-0500 mon.alpha-cen [ERR] pg 2.3 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [11,9]
  113. 2024-03-08T23:40:00.000386-0500 mon.alpha-cen [ERR] pg 2.9 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [5,4]
  114. 2024-03-08T23:40:00.000409-0500 mon.alpha-cen [ERR] pg 2.11 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [4,9]
  115. 2024-03-08T23:40:00.000425-0500 mon.alpha-cen [ERR] pg 2.12 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [9,11]
  116. 2024-03-08T23:40:00.000440-0500 mon.alpha-cen [ERR] pg 2.14 is stuck undersized for 7m, current state active+undersized+degraded+remapped+backfilling, last acting [4,0]
  117. 2024-03-08T23:40:00.000460-0500 mon.alpha-cen [ERR] pg 2.1b is stuck undersized for 7m, current state active+undersized+degraded+remapped+inconsistent+backfilling, last acting [0,11]
  118. 2024-03-08T23:40:00.000476-0500 mon.alpha-cen [ERR] pg 2.1d is stuck undersized for 7m, current state active+undersized+degraded+remapped+backfilling, last acting [0,11]
  119. 2024-03-08T23:40:00.000493-0500 mon.alpha-cen [ERR] pg 4.8 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [9,14]
  120. 2024-03-08T23:40:00.000509-0500 mon.alpha-cen [ERR] pg 4.b is stuck undersized for 7m, current state active+undersized+degraded+remapped+backfilling, last acting [0,1]
  121. 2024-03-08T23:40:00.000528-0500 mon.alpha-cen [ERR] pg 4.12 is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [9,11]
  122. 2024-03-08T23:40:00.000587-0500 mon.alpha-cen [ERR] pg 4.1f is stuck undersized for 12m, current state active+undersized+degraded+remapped+backfilling, last acting [8,9]
  123. 2024-03-08T23:40:03.819415-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 64310/2020632 objects degraded (3.183%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  124. 2024-03-08T23:40:08.821296-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 63819/2020632 objects degraded (3.158%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  125. 2024-03-08T23:40:13.823400-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 63552/2020632 objects degraded (3.145%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  126. 2024-03-08T23:40:18.825790-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 63076/2020632 objects degraded (3.122%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  127. 2024-03-08T23:40:23.827980-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 62766/2020632 objects degraded (3.106%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  128. 2024-03-08T23:40:28.830062-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 62236/2020632 objects degraded (3.080%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  129. 2024-03-08T23:40:33.832482-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 61917/2020632 objects degraded (3.064%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  130. 2024-03-08T23:40:38.834563-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 61481/2020632 objects degraded (3.043%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  131. 2024-03-08T23:40:43.837046-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 61139/2020632 objects degraded (3.026%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  132. 2024-03-08T23:40:48.839306-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 60662/2020632 objects degraded (3.002%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  133. 2024-03-08T23:40:53.841394-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 60333/2020632 objects degraded (2.986%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  134. 2024-03-08T23:40:58.844239-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 60007/2020632 objects degraded (2.970%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  135. 2024-03-08T23:41:03.847031-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 59504/2020632 objects degraded (2.945%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  136. 2024-03-08T23:41:08.849388-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 59245/2020632 objects degraded (2.932%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  137. 2024-03-08T23:41:13.851663-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 58765/2020632 objects degraded (2.908%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  138. 2024-03-08T23:41:18.854351-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 58586/2020632 objects degraded (2.899%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  139. 2024-03-08T23:41:23.857322-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 57988/2020632 objects degraded (2.870%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  140. 2024-03-08T23:41:28.859590-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 57739/2020632 objects degraded (2.857%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  141. 2024-03-08T23:41:33.861729-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 57335/2020632 objects degraded (2.837%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  142. 2024-03-08T23:41:38.864094-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 56863/2020632 objects degraded (2.814%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  143. 2024-03-08T23:41:43.866333-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 56526/2020632 objects degraded (2.797%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  144. 2024-03-08T23:41:48.868705-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 56186/2020632 objects degraded (2.781%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  145. 2024-03-08T23:41:53.872172-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 55846/2020632 objects degraded (2.764%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  146. 2024-03-08T23:41:58.874312-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 55419/2020632 objects degraded (2.743%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  147. 2024-03-08T23:42:03.876680-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 55065/2020632 objects degraded (2.725%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  148. 2024-03-08T23:42:08.878785-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 54827/2020632 objects degraded (2.713%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  149. 2024-03-08T23:42:13.880997-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 54282/2020632 objects degraded (2.686%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  150. 2024-03-08T23:42:18.883164-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 54087/2020632 objects degraded (2.677%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  151. 2024-03-08T23:42:23.885550-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 53477/2020632 objects degraded (2.647%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  152. 2024-03-08T23:42:28.887685-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 53239/2020635 objects degraded (2.635%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  153. 2024-03-08T23:42:33.890548-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 52814/2020635 objects degraded (2.614%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  154. 2024-03-08T23:42:38.892705-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 52391/2020635 objects degraded (2.593%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  155. 2024-03-08T23:42:43.895348-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 51924/2020635 objects degraded (2.570%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  156. 2024-03-08T23:42:48.897585-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 51546/2020635 objects degraded (2.551%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  157. 2024-03-08T23:42:53.899731-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 51082/2020635 objects degraded (2.528%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  158. 2024-03-08T23:42:58.901929-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 50665/2020635 objects degraded (2.507%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  159. 2024-03-08T23:43:03.904425-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 50224/2020635 objects degraded (2.486%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  160. 2024-03-08T23:43:08.906736-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 49837/2020635 objects degraded (2.466%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  161. 2024-03-08T23:43:13.908933-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 49430/2020635 objects degraded (2.446%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED)
  162. 2024-03-08T23:43:18.911007-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 49147/2020635 objects degraded (2.432%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  163. 2024-03-08T23:43:23.913209-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 49010/2020635 objects degraded (2.425%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  164. 2024-03-08T23:43:28.915278-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 48734/2020635 objects degraded (2.412%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  165. 2024-03-08T23:43:33.917668-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 48452/2020635 objects degraded (2.398%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  166. 2024-03-08T23:43:38.919812-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 48251/2020635 objects degraded (2.388%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  167. 2024-03-08T23:43:43.922001-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 47967/2020635 objects degraded (2.374%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  168. 2024-03-08T23:43:48.924314-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 47734/2020635 objects degraded (2.362%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  169. 2024-03-08T23:43:53.926547-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 47482/2020635 objects degraded (2.350%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  170. 2024-03-08T23:43:58.929176-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 47238/2020635 objects degraded (2.338%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  171. 2024-03-08T23:44:03.932197-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 46917/2020635 objects degraded (2.322%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  172. 2024-03-08T23:44:08.934276-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 46653/2020635 objects degraded (2.309%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  173. 2024-03-08T23:44:13.936385-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 46381/2020635 objects degraded (2.295%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  174. 2024-03-08T23:44:18.938698-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 46200/2020635 objects degraded (2.286%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED)
  175. 2024-03-08T23:44:28.946656-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 45690/2020635 objects degraded (2.261%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  176. 2024-03-08T23:44:33.949090-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 45507/2020635 objects degraded (2.252%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  177. 2024-03-08T23:44:38.951185-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 45429/2020635 objects degraded (2.248%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  178. 2024-03-08T23:44:43.953423-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 45214/2020635 objects degraded (2.238%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  179. 2024-03-08T23:44:48.955569-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44977/2020635 objects degraded (2.226%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  180. 2024-03-08T23:44:53.957638-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44666/2020635 objects degraded (2.210%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  181. 2024-03-08T23:44:58.959708-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44625/2020635 objects degraded (2.208%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  182. 2024-03-08T23:45:03.961985-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44439/2020638 objects degraded (2.199%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  183. 2024-03-08T23:45:08.964223-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44300/2020638 objects degraded (2.192%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  184. 2024-03-08T23:45:13.968988-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 44055/2020638 objects degraded (2.180%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  185. 2024-03-08T23:45:18.972134-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 43787/2020638 objects degraded (2.167%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  186. 2024-03-08T23:45:23.974199-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 43651/2020638 objects degraded (2.160%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  187. 2024-03-08T23:45:28.976336-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 43309/2020638 objects degraded (2.143%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  188. 2024-03-08T23:45:33.981174-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 43196/2020638 objects degraded (2.138%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  189. 2024-03-08T23:45:38.983656-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 42964/2020638 objects degraded (2.126%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  190. 2024-03-08T23:45:43.985863-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 42760/2020638 objects degraded (2.116%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  191. 2024-03-08T23:45:48.988055-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 42542/2020638 objects degraded (2.105%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  192. 2024-03-08T23:45:53.990202-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 42321/2020638 objects degraded (2.094%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  193. 2024-03-08T23:45:58.992363-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 42090/2020638 objects degraded (2.083%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  194. 2024-03-08T23:46:03.198219-0500 mon.alpha-cen [INF] osd.0 failed (root=default,host=gamma-cen) (connection refused reported by osd.8)
  195. 2024-03-08T23:46:03.994449-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 41845/2020638 objects degraded (2.071%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED)
  196. 2024-03-08T23:46:04.177630-0500 mon.alpha-cen [WRN] Health check failed: 1 osds down (OSD_DOWN)
  197. 2024-03-08T23:46:05.203422-0500 osd.1 [INF] 4.b continuing backfill to osd.16 from (9972'52551504,10014'52557940] 4:d56f42d4:::rbd_data.32649b576d39df.00000000000001fa:head to 10014'52557940
  198. 2024-03-08T23:46:05.204064-0500 osd.4 [INF] 2.14 continuing backfill to osd.17 from (9970'9554185,10014'9562530] 2:2e65dda4:::rbd_data.2246bf4ed242af.0000000000011102:head to 10014'9562530
  199. 2024-03-08T23:46:06.195803-0500 mon.alpha-cen [WRN] Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY)
  200. 2024-03-08T23:46:08.996553-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 137428/2020638 objects degraded (6.801%), 20 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  201. 2024-03-08T23:46:12.227124-0500 mon.alpha-cen [WRN] Health check update: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)
  202. 2024-03-08T23:46:13.998879-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 204592/2020638 objects degraded (10.125%), 33 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  203. 2024-03-08T23:46:19.001098-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 204409/2020638 objects degraded (10.116%), 33 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  204. 2024-03-08T23:46:24.003432-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 204213/2020638 objects degraded (10.106%), 33 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  205. 2024-03-08T23:46:26.286582-0500 mon.alpha-cen [INF] osd.11 failed (root=default,host=beta-cen) (connection refused reported by osd.13)
  206. 2024-03-08T23:46:27.282067-0500 mon.alpha-cen [WRN] Health check update: 2 osds down (OSD_DOWN)
  207. 2024-03-08T23:46:29.005654-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 204046/2020638 objects degraded (10.098%), 33 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  208. 2024-03-08T23:46:28.525421-0500 osd.9 [INF] 4.12 continuing backfill to osd.6 from (9973'46183004,10016'46189912] 4:4abadcc2:::rbd_data.2e006d89682c5b.0000000000003678:head to 10016'46189912
  209. 2024-03-08T23:46:28.527274-0500 osd.9 [INF] 2.12 continuing backfill to osd.15 from (9971'10223194,10014'10230054] 2:495242ec:::rbd_data.2246bf4ed242af.0000000000022790:head to 10014'10230054
  210. 2024-03-08T23:46:33.324159-0500 mon.alpha-cen [WRN] Health check update: Reduced data availability: 2 pgs inactive, 2 pgs down (PG_AVAILABILITY)
  211. 2024-03-08T23:46:33.324202-0500 mon.alpha-cen [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 1 scrub errors)
  212. 2024-03-08T23:46:33.324223-0500 mon.alpha-cen [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent)
  213. 2024-03-08T23:46:34.008004-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 312676/1981203 objects degraded (15.782%), 49 pgs degraded, 4 pgs undersized (PG_DEGRADED)
  214. 2024-03-08T23:46:39.010684-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 346312/1981203 objects degraded (17.480%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  215. 2024-03-08T23:46:44.012872-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 346105/1981203 objects degraded (17.469%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  216. 2024-03-08T23:46:49.015126-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 345911/1981203 objects degraded (17.460%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  217. 2024-03-08T23:46:54.017498-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 345728/1981203 objects degraded (17.450%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  218. 2024-03-08T23:46:59.019751-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 345569/1981203 objects degraded (17.442%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  219. 2024-03-08T23:47:04.022021-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 345390/1981203 objects degraded (17.433%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  220. 2024-03-08T23:47:04.328673-0500 mon.alpha-cen [WRN] Health check update: 1 osds down (OSD_DOWN)
  221. 2024-03-08T23:47:04.337202-0500 mon.alpha-cen [INF] osd.0 [v2:10.4.0.3:6816/4210248776,v1:10.4.0.3:6817/4210248776] boot
  222. 2024-03-08T23:47:05.332152-0500 mon.alpha-cen [WRN] Health check update: Reduced data availability: 5 pgs inactive, 2 pgs down (PG_AVAILABILITY)
  223. 2024-03-08T23:47:05.338655-0500 osd.0 [INF] 4.b continuing backfill to osd.16 from (9972'52551504,10014'52557940] 4:d650fc01:::rbd_data.29b7c58be52e5b.000000000000257c:head to 10014'52557940
  224. 2024-03-08T23:47:05.340186-0500 osd.4 [INF] 2.14 continuing backfill to osd.17 from (9970'9554185,10014'9562530] 2:2eba657a:::rbd_data.2246bf4ed242af.000000000001fb6d:head to 10014'9562530
  225. 2024-03-08T23:47:06.658076-0500 osd.0 [INF] 2.1b continuing backfill to osd.6 from (9971'17067184,10014'17073660] 2:dc1332a8:::rbd_data.99d921d8edc910.0000000000000198:head to 10014'17073660
  226. 2024-03-08T23:47:06.667198-0500 osd.0 [INF] 2.1d continuing backfill to osd.6 from (9972'32706188,10014'32712589] 2:bc28647d:::rbd_data.307ae0ca08e035.0000000000001d0a:head to 10014'32712589
  227. 2024-03-08T23:47:09.024236-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 267759/1981203 objects degraded (13.515%), 43 pgs degraded, 23 pgs undersized (PG_DEGRADED)
  228. 2024-03-08T23:47:09.788988-0500 mon.alpha-cen [INF] osd.0 failed (root=default,host=gamma-cen) (connection refused reported by osd.6)
  229. 2024-03-08T23:47:10.368644-0500 mon.alpha-cen [WRN] Health check update: 2 osds down (OSD_DOWN)
  230. 2024-03-08T23:47:11.384189-0500 mon.alpha-cen [ERR] Health check failed: 1 scrub errors (OSD_SCRUB_ERRORS)
  231. 2024-03-08T23:47:11.384261-0500 mon.alpha-cen [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED)
  232. 2024-03-08T23:47:11.384291-0500 mon.alpha-cen [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs down)
  233. 2024-03-08T23:47:11.388455-0500 osd.1 [INF] 4.b continuing backfill to osd.16 from (9972'52551504,10022'52557959] 4:d6546548:::rbd_data.e3e4d53a30958a.00000000000002f5:head to 10022'52557959
  234. 2024-03-08T23:47:11.391913-0500 osd.4 [INF] 2.14 continuing backfill to osd.17 from (9970'9554185,10022'9562545] 2:2ec2aeea:::rbd_data.2246bf4ed242af.00000000000786a8:head to 10022'9562545
  235. 2024-03-08T23:47:13.394423-0500 mon.alpha-cen [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 2 pgs down, 1 pg peering (PG_AVAILABILITY)
  236. 2024-03-08T23:47:13.394473-0500 mon.alpha-cen [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 1 scrub errors)
  237. 2024-03-08T23:47:13.394495-0500 mon.alpha-cen [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent)
  238. 2024-03-08T23:47:14.026730-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 156369/1981206 objects degraded (7.893%), 20 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  239. 2024-03-08T23:47:19.028898-0500 mon.alpha-cen [WRN] Health check update: Reduced data availability: 2 pgs inactive, 2 pgs down (PG_AVAILABILITY)
  240. 2024-03-08T23:47:19.028958-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 344890/1981206 objects degraded (17.408%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  241. 2024-03-08T23:47:22.046472-0500 mon.alpha-cen [WRN] Health check update: 1 osds down (OSD_DOWN)
  242. 2024-03-08T23:47:22.098436-0500 mon.alpha-cen [INF] osd.11 [v2:10.4.0.2:6800/3334230374,v1:10.4.0.2:6804/3334230374] boot
  243. 2024-03-08T23:47:23.129202-0500 osd.9 [INF] 4.12 continuing backfill to osd.6 from (9973'46183004,10016'46189912] 4:4ad6433c:::rbd_data.d89845df0f9caf.0000000000002313:head to 10016'46189912
  244. 2024-03-08T23:47:23.138400-0500 osd.9 [INF] 2.12 continuing backfill to osd.15 from (9971'10223194,10014'10230054] 2:496024d2:::rbd_data.ba9cb08d00b53a.00000000000013df:head to 10014'10230054
  245. 2024-03-08T23:47:24.031000-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 344706/1981206 objects degraded (17.399%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  246. 2024-03-08T23:47:24.104817-0500 osd.11 [INF] 2.1b continuing backfill to osd.6 from (9971'17067184,10014'17073660] 2:dc1332a8:::rbd_data.99d921d8edc910.0000000000000198:head to 10014'17073660
  247. 2024-03-08T23:47:24.104836-0500 osd.11 [INF] 2.1d continuing backfill to osd.6 from (9972'32706188,10014'32712589] 2:bc2864e8:::rbd_data.2246bf4ed242af.0000000000036dac:head to 10014'32712589
  248. 2024-03-08T23:47:25.223097-0500 mon.alpha-cen [INF] osd.11 failed (root=default,host=beta-cen) (connection refused reported by osd.5)
  249. 2024-03-08T23:47:27.207237-0500 osd.9 [INF] 2.12 continuing backfill to osd.15 from (9971'10223194,10028'10230060] 2:4960db00:::rbd_data.2246bf4ed242af.000000000001c5c3:head to 10028'10230060
  250. 2024-03-08T23:47:27.207564-0500 osd.9 [INF] 4.12 continuing backfill to osd.6 from (9973'46183104,10029'46189978] 4:4ad89ae3:::rbd_data.e3f9edf923f99f.0000000000001525:head to 10029'46189978
  251. 2024-03-08T23:47:28.145877-0500 mon.alpha-cen [WRN] Health check update: Reduced data availability: 3 pgs inactive, 2 pgs down (PG_AVAILABILITY)
  252. 2024-03-08T23:47:29.033113-0500 mon.alpha-cen [WRN] Health check update: 2 osds down (OSD_DOWN)
  253. 2024-03-08T23:47:29.033213-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 248533/1981218 objects degraded (12.544%), 42 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  254. 2024-03-08T23:47:34.035455-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 308935/1981218 objects degraded (15.593%), 47 pgs degraded, 3 pgs undersized (PG_DEGRADED)
  255. 2024-03-08T23:47:39.037597-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 344142/1981218 objects degraded (17.370%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  256. 2024-03-08T23:47:44.039813-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 343957/1981218 objects degraded (17.361%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
  257. 2024-03-08T23:47:49.041945-0500 mon.alpha-cen [WRN] Health check update: Degraded data redundancy: 343816/1981218 objects degraded (17.354%), 50 pgs degraded, 2 pgs undersized (PG_DEGRADED)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!