API tools faq
paste
alohamora

ceph pg problem

alohamora
Apr 23rd, 2014
171
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 37.76 KB | None | 0 0
raw download clone embed print report
  1. # ceph -w
  2. cluster 009d3518-e60d-4f74-a26d-c08c1976263c
  3. health HEALTH_WARN 224 pgs stuck unclean; recovery 277/142134 objects degraded (0.195%); clock skew detected on mon.storage0115-ib
  4. monmap e5: 5 mons at {storage0101-ib=10.0.0.101:6789/0,storage0107-ib=10.0.0.107:6789/0,storage0113-ib=10.0.0.113:6789/0,storage0114-ib=10.0.0.114:6789/0,storage0115-ib=10.0.0.115:6789/0}, election epoch 32, quorum 0,1,2,3,4 storage0101-ib,storage0107-ib,storage0113-ib,storage0114-ib,storage0115-ib
  5. mdsmap e11: 1/1/1 up {0=storage0106-ib=up:active}
  6. osdmap e989: 165 osds: 165 up, 165 in
  7. pgmap v2062: 10240 pgs, 5 pools, 160 GB data, 41208 objects
  8. 493 GB used, 447 TB / 448 TB avail
  9. 277/142134 objects degraded (0.195%)
  10. 224 active+remapped
  11. 10016 active+clean
  12.  
  13. 2014-04-23 15:27:34.512535 mon.0 [INF] pgmap v2061: 10240 pgs: 224 active+remapped, 10016 active+clean; 160 GB data, 493 GB used, 447 TB / 448 TB avail; 277/142134 objects degraded (0.195%)
  14.  
  15.  
  16.  
  17.  
  18.  
  19.  
  20.  
  21. #
  22.  
  23. [root@bmi-pocfe1 ~]# ceph pg repair 4.277
  24. instructing pg 4.277 on osd.50 to repair
  25. [root@bmi-pocfe1 ~]# ceph health detail
  26. HEALTH_WARN 224 pgs stuck unclean; recovery 277/142134 objects degraded (0.195%); clock skew detected on mon.storage0115-ib
  27. pg 4.d3 is stuck unclean for 2850.278010, current state active+remapped, last acting [157,136,138,14,80,2147483647,37,147,125,78,67,47,82,2147483647,112,31,109,38]
  28. pg 4.5a4 is stuck unclean for 3050.556490, current state active+remapped, last acting [49,119,125,116,27,30,152,2147483647,41,114,26,84,47,65,133,2147483647,35,63]
  29. pg 4.73d is stuck unclean for 2841.598329, current state active+remapped, last acting [77,2147483647,69,163,45,65,59,109,2147483647,0,122,162,57,39,130,113,37,143]
  30. pg 4.403 is stuck unclean for 2850.280923, current state active+remapped, last acting [131,157,76,75,108,80,127,2147483647,161,25,40,2147483647,156,70,35,61,87,78]
  31. pg 4.59f is stuck unclean for 2847.978157, current state active+remapped, last acting [16,150,24,155,127,106,44,133,163,2147483647,27,19,80,29,30,49,131,2147483647]
  32. pg 4.cb is stuck unclean for 3050.738728, current state active+remapped, last acting [74,117,63,24,132,108,33,122,124,160,153,125,149,128,2147483647,52,135,89]
  33. pg 4.cf is stuck unclean for 6276.078483, current state active+remapped, last acting [48,116,38,114,140,16,129,2147483647,60,91,90,145,5,124,88,49,2147483647,26]
  34. pg 4.ce is stuck unclean for 2854.599798, current state active+remapped, last acting [38,72,37,41,117,2147483647,118,29,48,34,130,150,55,32,152,80,2147483647,30]
  35. pg 4.728 is stuck unclean for 2843.752528, current state active+remapped, last acting [46,99,34,89,2147483647,20,140,125,36,38,108,50,39,97,8,25,2147483647,90]
  36. pg 4.c2 is stuck unclean for 1144.647202, current state active+remapped, last acting [76,128,18,103,50,23,158,22,86,139,84,90,30,0,2147483647,63,2147483647,120]
  37. pg 4.72e is stuck unclean for 2845.960710, current state active+remapped, last acting [96,58,115,125,112,80,27,6,2147483647,75,17,11,164,161,22,71,2147483647,18]
  38. pg 4.c1 is stuck unclean for 2845.952298, current state active+remapped, last acting [14,2147483647,2147483647,76,130,87,157,19,84,67,26,108,17,75,135,0,164,73]
  39. pg 4.25a is stuck unclean for 2845.765821, current state active+remapped, last acting [42,85,68,21,47,84,117,125,123,100,17,88,119,76,98,2147483647,79,2147483647]
  40. pg 4.594 is stuck unclean for 2854.461653, current state active+remapped, last acting [132,115,100,156,120,119,2147483647,69,1,108,136,39,48,129,125,2147483647,76,124]
  41. pg 4.722 is stuck unclean for 2850.260484, current state active+remapped, last acting [50,15,78,161,18,112,125,2147483647,44,11,77,149,71,121,40,2,145,2147483647]
  42. pg 4.c4 is stuck unclean for 2850.268393, current state active+remapped, last acting [122,105,78,50,124,142,1,87,103,2147483647,58,7,4,57,8,51,2147483647,135]
  43. pg 4.720 is stuck unclean for 2839.291743, current state active+remapped, last acting [4,143,125,91,136,2147483647,92,2147483647,154,119,133,144,40,28,138,33,53,115]
  44. pg 4.726 is stuck unclean for 2852.290582, current state active+remapped, last acting [10,83,137,135,110,130,117,95,86,151,145,88,12,3,44,111,2147483647,2147483647]
  45. pg 4.b9 is stuck unclean for 2839.295035, current state active+remapped, last acting [61,99,128,70,74,16,161,139,2147483647,79,52,75,146,162,2147483647,125,33,159]
  46. pg 4.725 is stuck unclean for 2848.127818, current state active+remapped, last acting [10,37,124,2,146,30,98,156,23,103,107,155,33,87,91,112,2147483647,2147483647]
  47. pg 4.b8 is stuck unclean for 2848.126406, current state active+remapped, last acting [138,88,135,62,81,155,149,2147483647,116,72,79,160,121,145,2147483647,68,4,18]
  48. pg 4.581 is stuck unclean for 2839.171689, current state active+remapped, last acting [79,69,8,102,42,59,1,141,33,122,2147483647,52,2147483647,135,58,22,146,66]
  49. pg 4.256 is stuck unclean for 2839.177306, current state active+remapped, last acting [93,89,14,92,10,7,84,38,41,33,39,146,32,125,95,2147483647,2147483647,103]
  50. pg 4.71d is stuck unclean for 2848.087452, current state active+remapped, last acting [62,7,70,35,2147483647,108,39,8,96,2147483647,30,86,125,69,32,28,155,97]
  51. pg 4.24e is stuck unclean for 2854.458639, current state active+remapped, last acting [24,92,121,142,128,148,4,103,2147483647,53,54,61,34,48,89,9,134,2147483647]
  52. pg 4.711 is stuck unclean for 6276.061272, current state active+remapped, last acting [78,160,40,5,2147483647,35,129,156,84,31,2147483647,36,135,149,37,114,98,89]
  53. pg 4.3e5 is stuck unclean for 6275.846064, current state active+remapped, last acting [63,125,100,26,81,19,18,162,16,57,72,20,105,75,2147483647,2147483647,129,22]
  54. pg 4.57d is stuck unclean for 2850.178550, current state active+remapped, last acting [160,125,134,46,39,142,29,90,65,2147483647,89,44,47,40,113,78,95,2147483647]
  55. pg 4.a9 is stuck unclean for 1134.899367, current state active+remapped, last acting [112,39,61,110,28,72,18,73,16,94,128,59,2147483647,2147483647,38,7,86,30]
  56. pg 4.241 is stuck unclean for 2848.078747, current state active+remapped, last acting [74,15,98,38,2147483647,4,129,155,61,2147483647,64,40,159,128,66,117,16,125]
  57. pg 4.714 is stuck unclean for 2854.595055, current state active+remapped, last acting [134,36,45,103,152,2147483647,43,14,136,48,79,117,2147483647,158,10,157,44,20]
  58. pg 4.70a is stuck unclean for 6288.774162, current state active+remapped, last acting [17,143,21,109,130,162,85,148,2147483647,163,71,31,126,60,2147483647,34,32,0]
  59. pg 4.a1 is stuck unclean for 1144.664262, current state active+remapped, last acting [120,76,159,54,145,126,89,150,2147483647,113,103,146,63,83,0,4,67,2147483647]
  60. pg 4.70d is stuck unclean for 2852.268729, current state active+remapped, last acting [133,56,114,75,2147483647,16,68,119,6,110,84,126,2147483647,125,121,112,65,97]
  61. pg 4.239 is stuck unclean for 2845.782823, current state active+remapped, last acting [130,136,17,142,16,58,163,47,161,2147483647,69,19,97,40,125,2147483647,89,141]
  62. pg 4.3d1 is stuck unclean for 6277.977610, current state active+remapped, last acting [48,21,139,147,70,44,66,101,77,99,76,2147483647,19,67,64,42,8,2147483647]
  63. pg 4.23f is stuck unclean for 2845.791455, current state active+remapped, last acting [53,42,95,67,49,9,162,2147483647,17,152,47,120,52,1,58,2147483647,136,12]
  64. pg 4.707 is stuck unclean for 1136.057207, current state active+remapped, last acting [74,108,20,52,124,122,2147483647,139,67,2,57,78,1,94,146,163,2147483647,73]
  65. pg 4.232 is stuck unclean for 2852.145458, current state active+remapped, last acting [88,98,61,11,64,2147483647,2147483647,12,14,89,160,36,70,59,110,53,133,24]
  66. pg 4.561 is stuck unclean for 2841.487845, current state active+remapped, last acting [14,137,10,0,1,121,50,47,40,2147483647,26,12,53,154,133,2147483647,34,143]
  67. pg 4.6ff is stuck unclean for 2839.259102, current state active+remapped, last acting [80,33,135,141,67,136,139,134,2147483647,57,30,34,88,125,58,2147483647,122,142]
  68. pg 4.92 is stuck unclean for 2854.583360, current state active+remapped, last acting [11,18,143,40,48,66,98,57,70,135,2147483647,39,138,69,21,72,14,2147483647]
  69. pg 4.3cc is stuck unclean for 5593.815178, current state active+remapped, last acting [48,107,142,119,132,75,115,14,127,2147483647,21,66,6,99,2147483647,84,104,133]
  70. pg 4.565 is stuck unclean for 2847.931469, current state active+remapped, last acting [153,29,152,25,6,155,74,40,117,70,100,2147483647,2,18,1,2147483647,0,7]
  71. pg 4.22f is stuck unclean for 2852.179276, current state active+remapped, last acting [8,100,110,38,115,148,2147483647,16,156,157,2147483647,154,124,77,150,1,139,124]
  72. pg 4.559 is stuck unclean for 2843.663925, current state active+remapped, last acting [106,140,122,37,95,65,57,9,123,153,148,53,2147483647,136,12,56,113,2147483647]
  73. pg 4.553 is stuck unclean for 2852.167444, current state active+remapped, last acting [68,29,112,146,44,133,2147483647,108,7,125,163,76,2147483647,123,91,90,118,110]
  74. pg 4.8e is stuck unclean for 2845.944716, current state active+remapped, last acting [36,158,74,2147483647,120,114,17,42,88,94,163,154,84,157,1,39,92,2147483647]
  75. pg 4.3b8 is stuck unclean for 3050.609834, current state active+remapped, last acting [149,159,66,26,115,10,2147483647,5,156,63,73,3,76,1,143,2147483647,77,56]
  76. pg 4.6e9 is stuck unclean for 2854.518046, current state active+remapped, last acting [139,122,126,141,54,48,43,77,125,2147483647,2147483647,10,57,40,143,30,145,81]
  77. pg 4.3bd is stuck unclean for 2848.120414, current state active+remapped, last acting [69,156,142,115,155,76,58,2147483647,44,12,32,27,101,43,23,138,2147483647,35]
  78. pg 4.3b3 is stuck unclean for 2848.095000, current state active+remapped, last acting [50,88,104,145,151,144,51,95,2147483647,101,109,125,155,107,2147483647,108,150,86]
  79. pg 4.3b2 is stuck unclean for 2839.241251, current state active+remapped, last acting [109,31,33,139,132,2147483647,45,24,38,9,98,142,67,32,2147483647,150,83,73]
  80. pg 4.86 is stuck unclean for 2850.207493, current state active+remapped, last acting [102,157,162,98,2147483647,100,60,163,69,37,121,141,2147483647,125,78,92,6,116]
  81. pg 4.6e2 is stuck unclean for 2841.570405, current state active+remapped, last acting [25,56,61,55,123,86,38,0,117,19,2147483647,92,164,49,66,71,2147483647,122]
  82. pg 4.6e0 is stuck unclean for 2850.269692, current state active+remapped, last acting [128,72,23,55,158,41,105,123,32,2147483647,126,152,88,2147483647,78,0,80,149]
  83. pg 4.3b5 is stuck unclean for 2843.745905, current state active+remapped, last acting [129,42,53,162,80,140,139,23,108,15,2147483647,65,54,125,2147483647,105,47,107]
  84. pg 4.6e7 is stuck unclean for 1144.280702, current state active+remapped, last acting [142,27,122,60,54,148,115,42,2147483647,120,63,50,4,33,53,129,150,2147483647]
  85. pg 4.79 is stuck unclean for 2839.264139, current state active+remapped, last acting [57,104,10,158,37,95,124,143,32,100,125,5,33,2147483647,40,92,123,2147483647]
  86. pg 4.6e5 is stuck unclean for 2850.245533, current state active+remapped, last acting [160,13,34,152,78,39,135,154,15,2147483647,38,136,56,14,2147483647,26,58,67]
  87. pg 4.210 is stuck unclean for 2839.118576, current state active+remapped, last acting [13,37,160,89,54,56,99,161,74,47,2147483647,115,92,33,42,55,2147483647,8]
  88. pg 4.73 is stuck unclean for 2852.298081, current state active+remapped, last acting [93,157,60,59,47,44,125,2147483647,40,23,65,76,116,114,110,54,139,2147483647]
  89. pg 4.3ad is stuck unclean for 1135.986465, current state active+remapped, last acting [83,158,89,150,2,2147483647,151,163,117,137,43,2147483647,48,96,79,94,55,88]
  90. pg 4.545 is stuck unclean for 6291.725679, current state active+remapped, last acting [140,137,86,21,3,134,27,83,91,109,19,54,2147483647,50,138,139,2147483647,133]
  91. pg 4.3a3 is stuck unclean for 3050.998564, current state active+remapped, last acting [77,58,139,44,14,2147483647,123,63,30,51,130,115,37,136,11,118,2147483647,134]
  92. pg 4.70 is stuck unclean for 2848.115402, current state active+remapped, last acting [19,46,83,4,154,150,50,155,24,104,20,111,27,133,2147483647,72,2147483647,125]
  93. pg 4.53a is stuck unclean for 1135.689884, current state active+remapped, last acting [80,4,149,18,116,81,106,31,19,138,2,162,63,84,94,2147483647,2147483647,105]
  94. pg 4.20f is stuck unclean for 2850.124752, current state active+remapped, last acting [103,161,72,55,19,137,2147483647,106,78,108,2147483647,144,104,86,138,43,74,60]
  95. pg 4.20e is stuck unclean for 2854.439292, current state active+remapped, last acting [112,162,48,104,131,2,29,159,2147483647,154,51,92,50,125,45,106,150,2147483647]
  96. pg 4.6d7 is stuck unclean for 2854.549416, current state active+remapped, last acting [143,48,125,141,136,46,3,51,120,26,145,103,2147483647,160,70,101,2147483647,83]
  97. pg 4.39f is stuck unclean for 2852.236496, current state active+remapped, last acting [26,42,11,37,110,158,24,151,124,50,103,117,162,38,2147483647,2147483647,123,105]
  98. pg 4.6c9 is stuck unclean for 2845.928616, current state active+remapped, last acting [40,157,10,17,14,96,125,111,2147483647,39,7,122,100,112,32,79,2147483647,70]
  99. pg 4.6cf is stuck unclean for 2854.560356, current state active+remapped, last acting [116,135,24,48,31,90,125,100,25,2147483647,131,139,60,2147483647,39,104,88,108]
  100. pg 4.1fb is stuck unclean for 2841.402613, current state active+remapped, last acting [20,158,117,71,0,113,44,142,76,160,40,76,136,2147483647,26,2147483647,163,30]
  101. pg 4.6c2 is stuck unclean for 1134.815157, current state active+remapped, last acting [146,102,69,141,6,64,2147483647,19,118,138,2147483647,71,31,163,99,110,94,59]
  102. pg 4.1fc is stuck unclean for 2854.410821, current state active+remapped, last acting [133,153,76,58,3,100,121,128,2147483647,25,87,61,138,61,145,2147483647,48,147]
  103. pg 4.38e is stuck unclean for 2843.742856, current state active+remapped, last acting [61,152,164,83,39,126,125,2147483647,43,87,19,1,140,109,99,4,40,2147483647]
  104. pg 4.38d is stuck unclean for 2852.246068, current state active+remapped, last acting [61,45,27,35,148,18,156,115,4,110,2147483647,43,87,58,25,26,76,2147483647]
  105. pg 4.38c is stuck unclean for 2848.101806, current state active+remapped, last acting [123,29,154,31,40,149,44,50,125,59,2147483647,155,135,58,102,131,2147483647,139]
  106. pg 4.6bd is stuck unclean for 2848.038434, current state active+remapped, last acting [143,152,38,10,32,57,132,111,84,155,18,65,95,2147483647,2147483647,66,148,28]
  107. pg 4.1e9 is stuck unclean for 2845.845545, current state active+remapped, last acting [112,2,6,17,84,1,100,43,126,148,71,149,145,2147483647,73,2147483647,122,59]
  108. pg 4.51c is stuck unclean for 3050.735314, current state active+remapped, last acting [51,2147483647,63,13,134,125,10,131,24,88,37,62,66,71,127,120,2147483647,89]
  109. pg 4.6b5 is stuck unclean for 2845.852715, current state active+remapped, last acting [114,135,144,85,90,81,113,37,24,132,2147483647,19,60,2147483647,104,92,17,159]
  110. pg 4.511 is stuck unclean for 2841.448933, current state active+remapped, last acting [111,11,34,133,116,129,2147483647,0,38,9,74,2147483647,73,109,18,118,153,61]
  111. pg 4.516 is stuck unclean for 2843.602122, current state active+remapped, last acting [141,41,9,154,153,134,162,106,46,140,90,51,14,98,2147483647,125,101,2147483647]
  112. pg 4.37c is stuck unclean for 5589.547445, current state active+remapped, last acting [155,130,103,133,112,101,2147483647,129,90,2147483647,33,32,46,115,116,159,3,163]
  113. pg 4.41 is stuck unclean for 2850.234681, current state active+remapped, last acting [146,67,71,86,11,93,55,2147483647,15,2147483647,47,162,65,124,119,150,78,125]
  114. pg 4.6a3 is stuck unclean for 2850.206846, current state active+remapped, last acting [39,79,78,131,103,109,55,76,89,58,91,2147483647,95,2147483647,90,119,144,100]
  115. pg 4.370 is stuck unclean for 2848.084412, current state active+remapped, last acting [141,134,138,97,158,2147483647,155,147,2147483647,55,90,152,151,80,126,24,8,67]
  116. pg 4.1dd is stuck unclean for 2839.018682, current state active+remapped, last acting [97,50,69,58,122,87,19,40,37,5,124,2147483647,33,32,39,2147483647,1,10]
  117. pg 4.50f is stuck unclean for 2839.058501, current state active+remapped, last acting [44,101,134,129,2147483647,5,156,130,132,152,154,33,47,4,35,65,2147483647,125]
  118. pg 4.3b is stuck unclean for 2845.903888, current state active+remapped, last acting [4,66,154,8,2147483647,5,42,17,160,107,72,20,123,98,164,2147483647,70,58]
  119. pg 4.3a is stuck unclean for 2850.169707, current state active+remapped, last acting [14,119,78,60,9,19,20,2147483647,107,10,84,108,53,4,8,2147483647,23,113]
  120. pg 4.1d1 is stuck unclean for 2850.115791, current state active+remapped, last acting [5,145,60,50,144,147,25,76,4,74,78,2,61,2147483647,163,2147483647,64,125]
  121. pg 4.502 is stuck unclean for 2847.951785, current state active+remapped, last acting [37,119,86,151,56,52,66,155,83,16,10,2147483647,72,2147483647,148,125,15,4]
  122. pg 4.1d7 is stuck unclean for 2847.899748, current state active+remapped, last acting [153,90,77,101,34,155,50,53,40,66,83,149,68,39,11,142,2147483647,2147483647]
  123. pg 4.30 is stuck unclean for 2852.212736, current state active+remapped, last acting [70,59,85,2147483647,61,46,71,2147483647,162,53,125,123,154,24,110,80,39,82]
  124. pg 4.37 is stuck unclean for 2852.257656, current state active+remapped, last acting [146,73,54,110,122,2147483647,29,151,2147483647,125,123,64,102,90,65,4,129,109]
  125. pg 4.4f9 is stuck unclean for 2839.166520, current state active+remapped, last acting [103,28,115,1,159,87,2147483647,125,120,62,33,67,74,2147483647,4,24,97,126]
  126. pg 4.691 is stuck unclean for 2852.246858, current state active+remapped, last acting [118,54,13,91,126,9,58,101,112,90,2147483647,31,110,125,148,42,145,2147483647]
  127. pg 4.2a is stuck unclean for 2845.856401, current state active+remapped, last acting [29,17,163,23,82,85,3,125,72,40,136,58,2147483647,151,60,107,2147483647,130]
  128. pg 4.1c1 is stuck unclean for 2839.180246, current state active+remapped, last acting [49,129,69,57,96,2147483647,119,114,67,101,33,90,47,158,46,52,2147483647,103]
  129. pg 4.681 is stuck unclean for 2852.281084, current state active+remapped, last acting [138,110,108,125,53,163,151,74,24,2147483647,152,114,65,147,14,4,161,2147483647]
  130. pg 4.1be is stuck unclean for 2845.751885, current state active+remapped, last acting [58,161,10,89,57,22,2147483647,125,11,64,2147483647,19,96,47,56,26,80,17]
  131. pg 4.4ee is stuck unclean for 3050.492361, current state active+remapped, last acting [126,36,19,43,76,84,42,134,5,121,2147483647,118,2147483647,6,68,75,125,63]
  132. pg 4.1b1 is stuck unclean for 3050.641106, current state active+remapped, last acting [30,115,93,73,143,4,161,43,118,2147483647,150,83,120,16,53,119,2147483647,63]
  133. pg 4.34e is stuck unclean for 2848.092873, current state active+remapped, last acting [69,103,155,150,18,107,134,100,68,2147483647,135,2147483647,140,42,102,124,1,19]
  134. pg 4.4d8 is stuck unclean for 1139.274633, current state active+remapped, last acting [121,47,118,60,28,31,159,68,71,81,2147483647,140,112,63,7,3,2147483647,87]
  135. pg 4.14 is stuck unclean for 2848.032099, current state active+remapped, last acting [80,155,152,147,64,97,41,24,47,84,2147483647,21,148,111,70,115,2147483647,26]
  136. pg 4.346 is stuck unclean for 3050.755748, current state active+remapped, last acting [81,162,125,53,24,4,32,111,164,71,3,115,62,134,130,2147483647,2147483647,63]
  137. pg 4.66a is stuck unclean for 2850.242714, current state active+remapped, last acting [36,57,81,2,66,45,96,126,78,62,147,115,2147483647,125,6,104,2147483647,121]
  138. pg 4.4d1 is stuck unclean for 2854.511762, current state active+remapped, last acting [156,118,53,69,117,48,98,59,80,6,2147483647,86,62,54,2147483647,49,75,87]
  139. pg 4.1a6 is stuck unclean for 2847.953045, current state active+remapped, last acting [143,6,2147483647,46,55,74,147,34,159,126,155,67,57,43,2147483647,54,45,67]
  140. pg 4.1a5 is stuck unclean for 5591.226974, current state active+remapped, last acting [17,46,103,129,71,101,12,100,4,72,10,82,2147483647,91,2147483647,90,81,18]
  141. pg 4.33c is stuck unclean for 2845.834349, current state active+remapped, last acting [90,84,2147483647,79,22,145,8,101,2147483647,39,91,28,17,143,127,128,125,62]
  142. pg 4.198 is stuck unclean for 2847.959890, current state active+remapped, last acting [97,103,74,65,64,117,127,45,151,155,70,137,146,2147483647,96,157,2147483647,77]
  143. pg 4.19f is stuck unclean for 3050.664514, current state active+remapped, last acting [10,38,63,20,1,83,25,154,128,66,148,27,2147483647,115,2147483647,31,99,36]
  144. pg 4.19e is stuck unclean for 1144.626411, current state active+remapped, last acting [9,46,63,108,13,88,71,105,54,0,161,123,91,143,52,2147483647,2147483647,26]
  145. pg 4.660 is stuck unclean for 2850.214265, current state active+remapped, last acting [87,116,161,45,78,113,159,120,114,131,105,18,103,31,90,69,2147483647,2147483647]
  146. pg 4.4ce is stuck unclean for 2852.118590, current state active+remapped, last acting [25,16,19,100,64,110,120,143,90,89,70,67,2147483647,14,162,156,36,2147483647]
  147. pg 4.193 is stuck unclean for 1135.775331, current state active+remapped, last acting [100,78,11,64,128,145,38,94,1,35,129,2147483647,2147483647,133,109,55,40,20]
  148. pg 4.32b is stuck unclean for 2854.554631, current state active+remapped, last acting [83,161,106,15,132,87,34,154,2147483647,35,48,43,85,159,90,2147483647,62,24]
  149. pg 4.664 is stuck unclean for 2845.833906, current state active+remapped, last acting [137,17,95,163,73,83,100,93,90,2147483647,2147483647,15,161,82,44,68,151,142]
  150. pg 4.7f3 is stuck unclean for 2852.155445, current state active+remapped, last acting [114,158,42,20,110,137,37,10,24,88,33,84,2147483647,55,2147483647,163,15,102]
  151. pg 4.197 is stuck unclean for 2845.736912, current state active+remapped, last acting [137,146,125,154,41,70,106,145,21,14,13,2147483647,4,2147483647,88,89,17,5]
  152. pg 4.4c5 is stuck unclean for 3050.823901, current state active+remapped, last acting [29,62,153,101,109,63,2147483647,52,2,134,21,61,108,81,146,27,2147483647,59]
  153. pg 4.65d is stuck unclean for 2852.205054, current state active+remapped, last acting [157,80,164,126,145,2147483647,56,81,15,159,135,33,12,85,92,110,128,2147483647]
  154. pg 4.7ea is stuck unclean for 2845.808407, current state active+remapped, last acting [4,7,25,17,128,18,105,20,152,80,120,70,102,2147483647,147,41,89,2147483647]
  155. pg 4.326 is stuck unclean for 2850.200202, current state active+remapped, last acting [96,152,43,65,102,6,138,4,37,104,21,24,97,99,78,2147483647,132,2147483647]
  156. pg 4.4bf is stuck unclean for 6283.417718, current state active+remapped, last acting [78,108,158,53,62,147,5,99,83,74,75,12,2147483647,20,2147483647,114,61,26]
  157. pg 4.18c is stuck unclean for 2852.156250, current state active+remapped, last acting [28,41,139,117,128,122,111,104,27,110,49,91,82,2147483647,140,72,2147483647,105]
  158. pg 4.181 is stuck unclean for 2850.188620, current state active+remapped, last acting [138,23,90,118,78,86,98,124,126,83,127,106,114,38,2147483647,2147483647,62,83]
  159. pg 4.31a is stuck unclean for 2845.823226, current state active+remapped, last acting [76,159,83,154,130,164,151,138,102,69,41,73,163,17,19,2147483647,2147483647,119]
  160. pg 4.654 is stuck unclean for 3050.742914, current state active+remapped, last acting [18,63,62,111,69,144,20,129,148,162,98,124,2147483647,130,144,164,134,2147483647]
  161. pg 4.4b0 is stuck unclean for 3050.653329, current state active+remapped, last acting [10,162,135,2147483647,138,100,68,2147483647,104,96,142,19,153,63,16,128,117,137]
  162. pg 4.7e0 is stuck unclean for 2854.503483, current state active+remapped, last acting [60,131,59,31,162,48,50,55,2,2147483647,61,127,2147483647,39,149,83,141,154]
  163. pg 4.7e7 is stuck unclean for 2852.139721, current state active+remapped, last acting [92,134,29,88,153,86,21,49,2147483647,30,110,146,161,124,7,2147483647,125,20]
  164. pg 4.179 is stuck unclean for 2850.105906, current state active+remapped, last acting [121,8,2147483647,112,23,131,114,104,125,143,92,45,78,154,61,29,2147483647,96]
  165. pg 4.7e4 is stuck unclean for 2854.428530, current state active+remapped, last acting [112,151,150,2,159,29,28,48,62,2147483647,18,1,163,81,33,145,2147483647,118]
  166. pg 4.4a9 is stuck unclean for 2847.999436, current state active+remapped, last acting [107,141,41,69,14,106,100,3,101,121,155,134,2147483647,148,140,8,2147483647,131]
  167. pg 4.7db is stuck unclean for 2845.732015, current state active+remapped, last acting [131,114,89,96,41,85,77,29,93,11,2147483647,17,140,103,2147483647,117,90,119]
  168. pg 4.316 is stuck unclean for 2843.679996, current state active+remapped, last acting [153,135,81,58,140,2147483647,154,45,73,116,16,59,73,104,2147483647,82,72,80]
  169. pg 4.315 is stuck unclean for 2843.682727, current state active+remapped, last acting [144,24,162,39,45,130,131,104,151,150,152,2147483647,140,125,3,13,55,2147483647]
  170. pg 4.314 is stuck unclean for 2845.795261, current state active+remapped, last acting [42,35,118,68,62,87,44,17,32,2147483647,111,153,145,125,49,146,2147483647,161]
  171. pg 4.4a1 is stuck unclean for 1135.812675, current state active+remapped, last acting [41,137,66,26,45,88,2147483647,37,128,13,121,63,2147483647,134,19,135,67,94]
  172. pg 4.169 is stuck unclean for 1139.330601, current state active+remapped, last acting [91,21,26,9,63,128,145,68,140,108,87,31,15,51,2147483647,2147483647,27,53]
  173. pg 4.632 is stuck unclean for 2839.174192, current state active+remapped, last acting [86,134,141,153,103,57,164,62,161,33,32,2147483647,15,19,79,160,2147483647,125]
  174. pg 4.49e is stuck unclean for 2850.101800, current state active+remapped, last acting [42,145,78,143,59,93,51,101,27,25,125,12,2147483647,107,61,100,127,2147483647]
  175. pg 4.7c8 is stuck unclean for 1135.840477, current state active+remapped, last acting [16,26,87,30,114,78,29,25,18,117,40,52,2147483647,94,77,121,2147483647,143]
  176. pg 4.635 is stuck unclean for 2845.809658, current state active+remapped, last acting [29,81,123,4,163,88,152,2147483647,43,126,41,164,135,17,52,125,2147483647,72]
  177. pg 4.7c3 is stuck unclean for 2843.635663, current state active+remapped, last acting [22,93,60,130,51,2,2147483647,84,152,135,2147483647,8,140,14,125,127,145,150]
  178. pg 4.629 is stuck unclean for 2839.263421, current state active+remapped, last acting [30,33,65,38,31,79,153,116,109,150,120,102,2147483647,125,129,73,160,2147483647]
  179. pg 4.628 is stuck unclean for 2852.226434, current state active+remapped, last acting [65,101,150,119,132,2147483647,110,23,2147483647,18,13,151,80,86,95,29,137,33]
  180. pg 4.62e is stuck unclean for 2845.873765, current state active+remapped, last acting [126,50,148,164,144,19,2147483647,86,2147483647,163,1,73,72,125,152,120,17,85]
  181. pg 4.7c7 is stuck unclean for 2850.120863, current state active+remapped, last acting [13,93,161,115,78,112,30,74,2147483647,103,138,134,99,32,107,2147483647,29,28]
  182. pg 4.159 is stuck unclean for 1144.545708, current state active+remapped, last acting [71,27,51,98,54,16,141,63,2147483647,150,24,69,18,0,160,15,120,2147483647]
  183. pg 4.7bb is stuck unclean for 2854.524394, current state active+remapped, last acting [158,114,90,138,87,133,139,120,48,132,29,2147483647,125,154,157,72,11,2147483647]
  184. pg 4.48e is stuck unclean for 2839.045768, current state active+remapped, last acting [45,33,141,29,125,102,98,151,41,21,95,2147483647,154,40,66,70,137,2147483647]
  185. pg 4.627 is stuck unclean for 2839.275179, current state active+remapped, last acting [8,82,26,90,99,93,29,148,77,9,33,2147483647,31,124,149,2147483647,125,101]
  186. pg 4.7be is stuck unclean for 1136.280817, current state active+remapped, last acting [63,151,145,50,94,2147483647,13,126,2147483647,31,56,123,38,112,99,57,164,148]
  187. pg 4.485 is stuck unclean for 2843.527742, current state active+remapped, last acting [71,58,84,133,2147483647,136,2147483647,82,138,79,107,27,79,85,47,104,140,130]
  188. pg 4.2e3 is stuck unclean for 2847.948693, current state active+remapped, last acting [133,58,18,74,107,104,84,14,28,111,4,155,2147483647,125,20,82,2147483647,162]
  189. pg 4.2e2 is stuck unclean for 2854.542703, current state active+remapped, last acting [156,87,24,85,11,138,79,35,147,125,82,2147483647,9,161,2147483647,61,48,84]
  190. pg 4.14f is stuck unclean for 1143.847502, current state active+remapped, last acting [92,65,133,66,126,41,2147483647,135,139,63,30,57,88,2147483647,164,33,54,72]
  191. pg 4.14e is stuck unclean for 3050.905033, current state active+remapped, last acting [8,112,84,116,63,136,131,9,41,92,39,125,101,2147483647,54,2147483647,120,49]
  192. pg 4.47f is stuck unclean for 2845.706233, current state active+remapped, last acting [164,115,79,88,30,33,143,6,100,26,54,13,137,2147483647,161,2147483647,17,144]
  193. pg 4.617 is stuck unclean for 2841.359599, current state active+remapped, last acting [126,120,21,97,69,0,159,2147483647,58,28,111,2147483647,131,62,40,72,134,102]
  194. pg 4.615 is stuck unclean for 1134.888953, current state active+remapped, last acting [154,103,87,16,110,43,86,135,84,136,123,42,131,2147483647,94,112,2147483647,102]
  195. pg 4.2df is stuck unclean for 2839.218709, current state active+remapped, last acting [12,126,60,125,135,13,148,143,118,164,36,16,29,82,2147483647,33,2147483647,66]
  196. pg 4.475 is stuck unclean for 2847.946975, current state active+remapped, last acting [120,111,24,116,117,74,127,38,155,100,147,3,2147483647,44,98,154,2147483647,11]
  197. pg 4.60c is stuck unclean for 2854.531645, current state active+remapped, last acting [137,28,72,48,142,51,118,85,86,120,52,111,90,29,2147483647,125,2147483647,54]
  198. pg 4.138 is stuck unclean for 3050.707243, current state active+remapped, last acting [46,5,111,65,156,105,69,141,132,28,125,49,151,104,86,2147483647,2147483647,63]
  199. pg 4.468 is stuck unclean for 2843.558496, current state active+remapped, last acting [61,22,64,85,2147483647,10,44,56,68,130,140,122,128,45,39,7,2147483647,116]
  200. pg 4.600 is stuck unclean for 3050.712105, current state active+remapped, last acting [59,107,128,50,2147483647,27,63,2,7,1,61,125,54,55,3,44,2147483647,5]
  201. pg 4.79f is stuck unclean for 2845.737722, current state active+remapped, last acting [11,88,135,105,66,17,138,158,159,18,128,2147483647,161,125,2147483647,85,113,54]
  202. pg 4.605 is stuck unclean for 2839.241844, current state active+remapped, last acting [40,13,153,42,161,54,80,10,152,33,91,82,19,64,2147483647,101,2147483647,143]
  203. pg 4.2cd is stuck unclean for 2854.531296, current state active+remapped, last acting [92,143,14,2147483647,9,89,56,48,140,98,2147483647,136,3,141,134,148,139,135]
  204. pg 4.2c2 is stuck unclean for 2850.136750, current state active+remapped, last acting [83,154,96,2147483647,19,69,78,23,24,56,2147483647,66,121,31,15,149,33,91]
  205. pg 4.45b is stuck unclean for 2839.106273, current state active+remapped, last acting [43,134,71,4,10,147,33,91,2147483647,84,98,45,102,108,122,24,2147483647,10]
  206. pg 4.12e is stuck unclean for 2852.143004, current state active+remapped, last acting [149,161,2147483647,131,14,100,46,89,37,91,125,110,2147483647,58,85,162,10,84]
  207. pg 4.458 is stuck unclean for 2850.175447, current state active+remapped, last acting [45,154,144,117,61,70,68,78,158,59,152,109,55,100,125,2147483647,88,2147483647]
  208. pg 4.78e is stuck unclean for 2850.145633, current state active+remapped, last acting [159,73,139,115,61,57,78,2147483647,11,5,21,3,24,111,2147483647,9,104,51]
  209. pg 4.2ba is stuck unclean for 2848.074551, current state active+remapped, last acting [1,79,7,104,92,68,29,27,32,153,105,2147483647,102,2147483647,33,155,96,160]
  210. pg 4.5e8 is stuck unclean for 2852.230446, current state active+remapped, last acting [151,110,126,140,129,119,100,53,51,139,2147483647,16,2147483647,111,42,98,59,8]
  211. pg 4.124 is stuck unclean for 2852.149955, current state active+remapped, last acting [130,69,109,18,74,29,2147483647,163,157,125,89,11,2147483647,56,110,77,7,81]
  212. pg 4.786 is stuck unclean for 2852.100571, current state active+remapped, last acting [11,112,5,156,56,72,164,111,115,91,110,37,129,134,2147483647,9,74,2147483647]
  213. pg 4.5ec is stuck unclean for 2847.955359, current state active+remapped, last acting [127,40,77,18,155,133,107,82,128,100,113,2147483647,33,60,2147483647,150,96,163]
  214. pg 4.784 is stuck unclean for 2843.575584, current state active+remapped, last acting [65,153,149,27,2147483647,125,80,84,140,135,90,30,64,128,76,5,2147483647,86]
  215. pg 4.77b is stuck unclean for 6276.010428, current state active+remapped, last acting [155,66,0,72,138,31,40,2147483647,154,10,156,133,99,87,82,20,2147483647,26]
  216. pg 4.2b7 is stuck unclean for 2845.825507, current state active+remapped, last acting [84,117,17,99,118,119,4,10,37,111,2147483647,115,31,106,43,22,2147483647,57]
  217. pg 4.44f is stuck unclean for 2854.438175, current state active+remapped, last acting [107,83,124,2147483647,80,101,65,2147483647,47,132,163,48,69,10,86,85,5,37]
  218. pg 4.44e is stuck unclean for 2852.114874, current state active+remapped, last acting [77,38,46,140,102,129,36,57,110,145,28,2147483647,164,84,89,142,2147483647,56]
  219. pg 4.5e7 is stuck unclean for 2852.167164, current state active+remapped, last acting [128,44,15,120,23,3,2147483647,9,10,56,65,36,125,41,110,152,119,78]
  220. pg 4.111 is stuck unclean for 3050.767897, current state active+remapped, last acting [29,53,162,126,129,60,152,28,63,104,161,7,2147483647,21,125,2147483647,153,106]
  221. pg 4.5db is stuck unclean for 2854.530839, current state active+remapped, last acting [92,59,53,89,19,57,48,151,2147483647,88,127,113,125,12,2147483647,93,66,148]
  222. pg 4.114 is stuck unclean for 2845.760715, current state active+remapped, last acting [59,23,82,42,50,17,153,62,56,94,38,2147483647,154,99,96,1,2147483647,20]
  223. pg 4.2a3 is stuck unclean for 2843.662151, current state active+remapped, last acting [163,123,150,124,99,79,77,11,23,128,65,2147483647,140,100,8,162,2147483647,85]
  224. pg 4.5dd is stuck unclean for 1135.910866, current state active+remapped, last acting [66,25,131,78,77,81,37,36,27,92,123,68,2147483647,2147483647,90,29,94,12]
  225. pg 4.439 is stuck unclean for 2852.099204, current state active+remapped, last acting [13,158,73,91,74,2147483647,85,156,81,111,110,152,33,2147483647,100,18,68,149]
  226. pg 4.76b is stuck unclean for 2852.169781, current state active+remapped, last acting [3,87,2,153,83,32,110,147,76,109,1,148,160,2147483647,140,7,164,2147483647]
  227. pg 4.2a5 is stuck unclean for 2843.670553, current state active+remapped, last acting [88,18,129,92,44,55,70,98,140,2147483647,101,121,80,101,39,144,106,2147483647]
  228. pg 4.103 is stuck unclean for 1135.858138, current state active+remapped, last acting [40,126,150,63,137,98,119,2147483647,127,42,47,24,67,138,50,94,159,2147483647]
  229. pg 4.76e is stuck unclean for 2841.353157, current state active+remapped, last acting [60,10,127,157,83,66,149,82,72,0,40,152,23,2147483647,2147483647,152,61,37]
  230. pg 4.432 is stuck unclean for 2845.740987, current state active+remapped, last acting [80,96,144,93,113,17,2147483647,88,140,2147483647,39,90,46,95,98,91,54,89]
  231. pg 4.5ca is stuck unclean for 2841.379569, current state active+remapped, last acting [139,74,102,38,82,61,43,0,122,159,71,147,2147483647,2147483647,15,64,143,84]
  232. pg 4.fd is stuck unclean for 2839.023285, current state active+remapped, last acting [20,4,33,60,73,67,123,22,9,18,59,107,99,159,2147483647,2147483647,42,115]
  233. pg 4.295 is stuck unclean for 2843.736975, current state active+remapped, last acting [58,16,124,156,125,108,140,32,35,114,67,88,158,15,2147483647,2147483647,85,54]
  234. pg 4.421 is stuck unclean for 2854.606580, current state active+remapped, last acting [107,72,5,153,2147483647,70,48,57,2147483647,104,19,75,143,114,126,121,135,162]
  235. pg 4.753 is stuck unclean for 2850.296589, current state active+remapped, last acting [150,15,85,156,114,40,44,78,65,2147483647,2147483647,163,129,37,89,5,46,47]
  236. pg 4.283 is stuck unclean for 2854.399684, current state active+remapped, last acting [26,86,2147483647,151,161,130,48,84,87,18,35,16,2147483647,1,7,29,56,5]
  237. pg 4.282 is stuck unclean for 3050.757058, current state active+remapped, last acting [88,137,41,90,145,135,51,128,64,2147483647,96,123,12,26,125,146,2147483647,63]
  238. pg 4.419 is stuck unclean for 2850.269865, current state active+remapped, last acting [77,9,93,36,124,76,38,2,56,115,117,60,0,2147483647,78,90,2147483647,144]
  239. pg 4.5b2 is stuck unclean for 2854.501112, current state active+remapped, last acting [102,37,15,56,158,48,122,149,96,2147483647,50,136,26,29,2147483647,68,135,55]
  240. pg 4.286 is stuck unclean for 2850.131234, current state active+remapped, last acting [142,20,5,78,41,99,14,1,31,141,2147483647,2147483647,34,162,100,35,38,36]
  241. pg 4.5b7 is stuck unclean for 2852.176618, current state active+remapped, last acting [149,58,66,153,32,2147483647,98,53,2147483647,18,150,162,101,133,31,110,33,97]
  242. pg 4.285 is stuck unclean for 2843.687346, current state active+remapped, last acting [34,148,96,108,74,162,125,20,140,157,104,100,46,89,139,2147483647,2147483647,14]
  243. pg 4.74e is stuck unclean for 2839.304738, current state active+remapped, last acting [123,28,18,131,153,90,8,71,25,109,149,129,2147483647,152,117,6,33,2147483647]
  244. pg 4.27b is stuck unclean for 2843.654790, current state active+remapped, last acting [104,107,151,140,45,131,144,90,126,73,157,43,44,39,125,2147483647,85,2147483647]
  245. pg 4.e1 is stuck unclean for 2852.301870, current state active+remapped, last acting [96,74,88,14,28,76,133,139,41,20,154,145,110,125,123,12,2147483647,2147483647]
  246. pg 4.5b4 is stuck unclean for 2847.948193, current state active+remapped, last acting [104,155,25,43,138,158,2147483647,153,2147483647,129,125,29,64,152,147,148,150,77]
  247. pg 4.412 is stuck unclean for 2854.575002, current state active+remapped, last acting [3,41,107,13,32,119,105,7,29,103,48,30,112,133,146,2147483647,125,2147483647]
  248. pg 4.743 is stuck unclean for 2848.130767, current state active+remapped, last acting [60,139,83,137,155,162,125,90,65,49,106,124,160,163,2147483647,5,2147483647,157]
  249. pg 4.272 is stuck unclean for 2843.640172, current state active+remapped, last acting [61,66,89,38,23,112,2147483647,11,8,14,34,157,84,140,2147483647,3,74,154]
  250. pg 4.277 is stuck unclean for 2845.781776, current state active+remapped, last acting [50,92,26,53,158,106,17,164,8,138,49,70,86,104,2147483647,161,2147483647,71]
  251. recovery 277/142134 objects degraded (0.195%)
  252. mon.storage0115-ib addr 10.0.0.115:6789/0 clock skew 0.192309s > max 0.15s (latency 0.00109328s)
  253. [root@bmi-pocfe1 ~]# ceph pg repair 4.272
  254. instructing pg 4.272 on osd.61 to repair
  255. [root@bmi-pocfe1 ~]#
  256. [root@bmi-pocfe1 ~]#
  257. [root@bmi-pocfe1 ~]#
  258. [root@bmi-pocfe1 ~]#
  259. [root@bmi-pocfe1 ~]# ceph pg 4.272 query
  260. 2014-04-23 15:41:28.903812 7f5eac5f6700 0 -- 10.0.0.254:0/1057732 >> 192.168.1.114:6800/50576 pipe(0x7f5eb0016f90 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5eb0017200).fault
  261.  
  262.  
  263.  
  264. #
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!