API tools faq
paste
Advertisement
alohamora

PG stuck unclean

alohamora
Mar 5th, 2014
406
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 9.88 KB | None | 0 0
raw download clone embed print report
  1. [root@storage0101-ib ~]# ceph health detail
  2. HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec; 1 osds have slow requests
  3. pg 2.1566 is stuck inactive for 620902.955622, current state stale+peering, last acting [126]
  4. pg 34.1546 is stuck inactive for 711320.437550, current state stale+peering, last acting [126]
  5. pg 29.154b is stuck inactive for 642706.324688, current state stale+peering, last acting [126]
  6. pg 2.dc2 is stuck inactive for 620856.839348, current state stale+peering, last acting [126]
  7. pg 29.da7 is stuck inactive for 647248.132606, current state stale+peering, last acting [126]
  8. pg 34.da2 is stuck inactive for 621280.382233, current state stale+peering, last acting [126]
  9. pg 2.1566 is stuck unclean for 620902.965006, current state stale+peering, last acting [126]
  10. pg 34.1546 is stuck unclean for 711320.446930, current state stale+peering, last acting [126]
  11. pg 29.154b is stuck unclean for 642706.334066, current state stale+peering, last acting [126]
  12. pg 2.dc2 is stuck unclean for 620856.848727, current state stale+peering, last acting [126]
  13. pg 29.da7 is stuck unclean for 647248.141985, current state stale+peering, last acting [126]
  14. pg 34.da2 is stuck unclean for 621280.391612, current state stale+peering, last acting [126]
  15. pg 2.1566 is stuck stale for 620790.268205, current state stale+peering, last acting [126]
  16. pg 34.1546 is stuck stale for 620790.268216, current state stale+peering, last acting [126]
  17. pg 29.154b is stuck stale for 620790.268182, current state stale+peering, last acting [126]
  18. pg 2.dc2 is stuck stale for 620790.264263, current state stale+peering, last acting [126]
  19. pg 29.da7 is stuck stale for 620790.264253, current state stale+peering, last acting [126]
  20. pg 34.da2 is stuck stale for 620790.264106, current state stale+peering, last acting [126]
  21. 2 ops are blocked > 134218 sec
  22. 2 ops are blocked > 134218 sec on osd.73
  23. 1 osds have slow requests
  24. [root@storage0101-ib ~]#
  25.  
  26.  
  27. [root@storage0101-ib ~]#
  28. [root@storage0101-ib ~]# ceph pg dump_stuck stale
  29. ok
  30. pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp
  31. 2.1566 35 0 0 0 146800640 2905 2905 stale+peering 2014-02-26 11:49:04.446584 7531'353007 7532:922517 [126] [126] 7531'352799 2014-02-26 00:39:48.470762 7442'352455 2014-02-22 00:38:49.791111
  32. 34.1546 0 0 0 0 0 123 123 stale+peering 2014-02-26 11:49:04.439918 7498'3917628 7532:9499161 [126] [126] 7498'3917628 2014-02-25 10:40:50.113125 7498'3917628 2014-02-24 10:40:47.053466
  33. 29.154b 343 0 0 0 1438646272 350 350 stale+peering 2014-02-26 11:49:04.447632 4055'350 7532:8332 [126] [126] 4055'350 2014-02-26 05:44:24.226703 4055'350 2014-02-22 05:16:38.690145
  34. 2.dc2 96 0 0 0 402653184 3001 3001 stale+peering 2014-02-26 11:49:04.385774 7531'336893 7532:908354 [126] [126] 7531'335704 2014-02-25 12:22:40.239960 7531'335704 2014-02-24 12:22:39.359164
  35. 29.da7 720 0 0 0 3019898880 724 724 stale+peering 2014-02-26 11:49:04.417613 4055'724 7532:5214 [126] [126] 4055'724 2014-02-26 04:28:42.418536 4055'724 2014-02-22 04:28:01.304189
  36. 34.da2 2 0 0 0 4204255 2241 2241 stale+peering 2014-02-26 11:49:04.360489 7498'3823277 7532:9411318 [126] [126] 7498'3823277 2014-02-26 11:41:30.168786 7329'3823276 2014-02-20 09:50:20.585870
  37.  
  38. [root@storage0101-ib ~]#
  39. [root@storage0101-ib ~]#
  40. [root@storage0101-ib ~]# ceph pg dump_stuck inactive
  41. ok
  42. pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp
  43. 2.1566 35 0 0 0 146800640 2905 2905 stale+peering 2014-02-26 11:49:04.446584 7531'353007 7532:922517 [126] [126] 7531'352799 2014-02-26 00:39:48.470762 7442'352455 2014-02-22 00:38:49.791111
  44. 34.1546 0 0 0 0 0 123 123 stale+peering 2014-02-26 11:49:04.439918 7498'3917628 7532:9499161 [126] [126] 7498'3917628 2014-02-25 10:40:50.113125 7498'3917628 2014-02-24 10:40:47.053466
  45. 29.154b 343 0 0 0 1438646272 350 350 stale+peering 2014-02-26 11:49:04.447632 4055'350 7532:8332 [126] [126] 4055'350 2014-02-26 05:44:24.226703 4055'350 2014-02-22 05:16:38.690145
  46. 2.dc2 96 0 0 0 402653184 3001 3001 stale+peering 2014-02-26 11:49:04.385774 7531'336893 7532:908354 [126] [126] 7531'335704 2014-02-25 12:22:40.239960 7531'335704 2014-02-24 12:22:39.359164
  47. 29.da7 720 0 0 0 3019898880 724 724 stale+peering 2014-02-26 11:49:04.417613 4055'724 7532:5214 [126] [126] 4055'724 2014-02-26 04:28:42.418536 4055'724 2014-02-22 04:28:01.304189
  48. 34.da2 2 0 0 0 4204255 2241 2241 stale+peering 2014-02-26 11:49:04.360489 7498'3823277 7532:9411318 [126] [126] 7498'3823277 2014-02-26 11:41:30.168786 7329'3823276 2014-02-20 09:50:20.585870
  49.  
  50. [root@storage0101-ib ~]# ceph pg dump_stuck unclean
  51. ok
  52. pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp
  53. 2.1566 35 0 0 0 146800640 2905 2905 stale+peering 2014-02-26 11:49:04.446584 7531'353007 7532:922517 [126] [126] 7531'352799 2014-02-26 00:39:48.470762 7442'352455 2014-02-22 00:38:49.791111
  54. 34.1546 0 0 0 0 0 123 123 stale+peering 2014-02-26 11:49:04.439918 7498'3917628 7532:9499161 [126] [126] 7498'3917628 2014-02-25 10:40:50.113125 7498'3917628 2014-02-24 10:40:47.053466
  55. 29.154b 343 0 0 0 1438646272 350 350 stale+peering 2014-02-26 11:49:04.447632 4055'350 7532:8332 [126] [126] 4055'350 2014-02-26 05:44:24.226703 4055'350 2014-02-22 05:16:38.690145
  56. 2.dc2 96 0 0 0 402653184 3001 3001 stale+peering 2014-02-26 11:49:04.385774 7531'336893 7532:908354 [126] [126] 7531'335704 2014-02-25 12:22:40.239960 7531'335704 2014-02-24 12:22:39.359164
  57. 29.da7 720 0 0 0 3019898880 724 724 stale+peering 2014-02-26 11:49:04.417613 4055'724 7532:5214 [126] [126] 4055'724 2014-02-26 04:28:42.418536 4055'724 2014-02-22 04:28:01.304189
  58. 34.da2 2 0 0 0 4204255 2241 2241 stale+peering 2014-02-26 11:49:04.360489 7498'3823277 7532:9411318 [126] [126] 7498'3823277 2014-02-26 11:41:30.168786 7329'3823276 2014-02-20 09:50:20.585870
  59.  
  60. [root@storage0101-ib ~]#
  61.  
  62.  
  63.  
  64.  
  65. [root@storage0101-ib ~]#
  66. [root@storage0101-ib ~]# ceph pg dump | grep stale
  67. dumped all in format plain
  68. 34.1546 0 0 0 0 0 123 123 stale+peering 2014-02-26 11:49:04.439918 7498'3917628 7532:9499161 [126] [126] 7498'3917628 2014-02-25 10:40:50.113125 7498'3917628 2014-02-24 10:40:47.053466
  69. 2.1566 35 0 0 0 146800640 2905 2905 stale+peering 2014-02-26 11:49:04.446584 7531'353007 7532:922517 [126] [126] 7531'352799 2014-02-26 00:39:48.470762 7442'352455 2014-02-22 00:38:49.791111
  70. 29.154b 343 0 0 0 1438646272 350 350 stale+peering 2014-02-26 11:49:04.447632 4055'350 7532:8332 [126] [126] 4055'350 2014-02-26 05:44:24.226703 4055'350 2014-02-22 05:16:38.690145
  71. 2.dc2 96 0 0 0 402653184 3001 3001 stale+peering 2014-02-26 11:49:04.385774 7531'336893 7532:908354 [126] [126] 7531'335704 2014-02-25 12:22:40.239960 7531'335704 2014-02-24 12:22:39.359164
  72. 29.da7 720 0 0 0 3019898880 724 724 stale+peering 2014-02-26 11:49:04.417613 4055'724 7532:5214 [126] [126] 4055'724 2014-02-26 04:28:42.418536 4055'724 2014-02-22 04:28:01.304189
  73. 34.da2 2 0 0 0 4204255 2241 2241 stale+peering 2014-02-26 11:49:04.360489 7498'3823277 7532:9411318 [126] [126] 7498'3823277 2014-02-26 11:41:30.168786 7329'3823276 2014-02-20 09:50:20.585870
  74. [root@storage0101-ib ~]#
  75.  
  76.  
  77.  
  78. [root@storage0101-ib ~]# ceph osd dump| grep size
  79. pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45
  80. pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0
  81. pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 4063 owner 0
  82. pool 29 'storage0112-ib' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 850 owner 0
  83. pool 34 'cephfs' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7501 owner 0
  84. pool 36 'pool-A' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 4054 owner 0
  85. pool 52 'benchmark-pool01' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7433 owner 0
  86. pool 53 'benchmark-pool02' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7434 owner 0
  87. pool 54 'benchmark-pool03' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7435 owner 0
  88. pool 55 'benchmark-pool04' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7436 owner 0
  89. pool 56 'benchmark-pool05' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7437 owner 0
  90. pool 57 'benchmark-pool06' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7438 owner 0
  91. pool 58 'benchmark-pool07' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7439 owner 0
  92. pool 59 'benchmark-pool08' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 6000 pgp_num 6000 last_change 7440 owner 0
  93. [root@storage0101-ib ~]#
  94.  
  95.  
  96. [root@storage0101-ib ~]# ceph status
  97. cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5
  98. health HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec
  99. monmap e6: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0106-ib=192.168.100.106:6789/0,storage0111-ib=192.168.100.111:6789/0}, election epoch 2836, quorum 0,1,2 storage0101-ib,storage0106-ib,storage0111-ib
  100. mdsmap e58: 1/1/1 up {0=storage0101-ib=up:active}
  101. osdmap e9621: 153 osds: 153 up, 153 in
  102. pgmap v1432289: 66256 pgs, 30 pools, 200 TB data, 51678 kobjects
  103. 90451 GB used, 319 TB / 416 TB avail
  104. 66250 active+clean
  105. 6 stale+peering
  106.  
  107. [root@storage0101-ib ~]#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!