API tools faq
paste
Advertisement
Guest User

Ceph OSD

a guest
Aug 11th, 2014
77
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 35.41 KB | None | 0 0
raw download clone embed print report
  1.   -126> 2014-08-11 22:58:33.429773 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(0.1c epoch 17542) v2 -- ?+0 0x1ced8e00 con 0x2228dc0
  2.   -125> 2014-08-11 22:58:33.429817 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[2.1e( v 15531'38 (0'0,15531'38] local-les=17529 n=1 ec=1 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  3.  -124> 2014-08-11 22:58:33.429831 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(2.1e epoch 17542) v2 -- ?+0 0x1ced8c00 con 0x2228dc0
  4.  -123> 2014-08-11 22:58:33.429875 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[4.18( empty local-les=17529 n=0 ec=9 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  5.   -122> 2014-08-11 22:58:33.429888 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(4.18 epoch 17542) v2 -- ?+0 0x1ced8a00 con 0x2228dc0
  6.   -121> 2014-08-11 22:58:33.429919 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[6.1d( v 17536'425743 (17139'422741,17536'425743] local-les=17530 n=4365 ec=15501 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17536 pi=17014-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  7.   -120> 2014-08-11 22:58:33.429932 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(6.1d(7) epoch 17542) v4 -- ?+0 0x1cebe000 con 0x2228dc0
  8.   -119> 2014-08-11 22:58:33.429970 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[1.1a( v 237'1247 (193'247,237'1247] local-les=17529 n=2 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  9.  -118> 2014-08-11 22:58:33.429988 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.1a epoch 17542) v2 -- ?+0 0x1ced8800 con 0x2228c60
  10.  -117> 2014-08-11 22:58:33.430026 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[0.1b( v 9147'4209 (237'3209,9147'4209] local-les=17529 n=12 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  11.   -116> 2014-08-11 22:58:33.430039 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(0.1b epoch 17542) v2 -- ?+0 0x1ced8600 con 0x2228c60
  12.   -115> 2014-08-11 22:58:33.430076 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[2.19( empty local-les=17529 n=0 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  13.  -114> 2014-08-11 22:58:33.430089 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(2.19 epoch 17542) v2 -- ?+0 0x1ced8400 con 0x2228c60
  14.  -113> 2014-08-11 22:58:33.430133 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[4.1f( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16786-17526/8 inactive NOTIFY] state<Start>: transitioning to Stray
  15.  -112> 2014-08-11 22:58:33.430147 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.1f(8) epoch 17542) v4 -- ?+0 0x1cedce00 con 0x2228dc0
  16.  -111> 2014-08-11 22:58:33.430192 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[1.1b( v 4656'2410 (260'1410,4656'2410] local-les=17529 n=3 ec=1 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  17.   -110> 2014-08-11 22:58:33.430211 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(1.1b epoch 17542) v2 -- ?+0 0x1ced8200 con 0x2228dc0
  18.   -109> 2014-08-11 22:58:33.430247 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[1.18( v 4705'423 (0'0,4705'423] local-les=17530 n=6 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  19.   -108> 2014-08-11 22:58:33.430259 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(1.18(7) epoch 17542) v4 -- ?+0 0x1cedcc40 con 0x2228dc0
  20.   -107> 2014-08-11 22:58:33.430294 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[0.19( v 9147'4464 (4670'3463,9147'4464] local-les=17530 n=11 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  21.   -106> 2014-08-11 22:58:33.430306 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(0.19(7) epoch 17542) v4 -- ?+0 0x1cedca80 con 0x2228dc0
  22.   -105> 2014-08-11 22:58:33.430364 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.15( v 193'5 (0'0,193'5] local-les=17530 n=4 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  23.   -104> 2014-08-11 22:58:33.430376 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(1.15(3) epoch 17542) v4 -- ?+0 0x1cecb380 con 0x2228c60
  24.   -103> 2014-08-11 22:58:33.430408 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.14( v 9147'10898 (2830'9898,9147'10898] local-les=17530 n=18 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  25.   -102> 2014-08-11 22:58:33.430420 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(0.14(3) epoch 17542) v4 -- ?+0 0x1cecb1c0 con 0x2228c60
  26.   -101> 2014-08-11 22:58:33.430460 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.16( empty local-les=17530 n=0 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  27.   -100> 2014-08-11 22:58:33.430472 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(2.16(7) epoch 17542) v4 -- ?+0 0x1cecb000 con 0x2228dc0
  28.    -99> 2014-08-11 22:58:33.430503 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.10( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 inactive NOTIFY] state<Start>: transitioning to Stray
  29.    -98> 2014-08-11 22:58:33.430517 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(4.10(3) epoch 17542) v4 -- ?+0 0x1cedee00 con 0x2228c60
  30.    -97> 2014-08-11 22:58:33.430560 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.15( v 17397'513170 (17393'510169,17397'513170] local-les=17529 n=4360 ec=15501 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  31.   -96> 2014-08-11 22:58:33.430579 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(6.15 epoch 17542) v2 -- ?+0 0x1ced2c00 con 0x2228c60
  32.   -95> 2014-08-11 22:58:33.430625 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.12( v 4707'437 (0'0,4707'437] local-les=17530 n=5 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  33.   -94> 2014-08-11 22:58:33.430638 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(1.12(12) epoch 17542) v4 -- ?+0 0x1cedec40 con 0x2228dc0
  34.   -93> 2014-08-11 22:58:33.430687 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.13( v 9147'8141 (4674'7141,9147'8141] local-les=17530 n=14 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  35.   -92> 2014-08-11 22:58:33.430699 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(0.13(12) epoch 17542) v4 -- ?+0 0x1cedea80 con 0x2228dc0
  36.   -91> 2014-08-11 22:58:33.430741 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.11( empty local-les=17530 n=0 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/12 inactive NOTIFY] state<Start>: transitioning to Stray
  37.   -90> 2014-08-11 22:58:33.430758 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(2.11(12) epoch 17542) v4 -- ?+0 0x1cede8c0 con 0x2228dc0
  38.   -89> 2014-08-11 22:58:33.430796 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.17( empty local-les=17529 n=0 ec=9 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  39.    -88> 2014-08-11 22:58:33.430809 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(4.17 epoch 17542) v2 -- ?+0 0x1ced2a00 con 0x2228c60
  40.    -87> 2014-08-11 22:58:33.430840 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.13( v 4705'574 (0'0,4705'574] local-les=17530 n=4 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  41.    -86> 2014-08-11 22:58:33.430856 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(1.13(3) epoch 17542) v4 -- ?+0 0x1cede700 con 0x2228c60
  42.    -85> 2014-08-11 22:58:33.430897 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.12( v 15734'18394 (4702'17394,15734'18394] local-les=17529 n=7 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  43.   -84> 2014-08-11 22:58:33.430911 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(0.12 epoch 17542) v2 -- ?+0 0x1ced2800 con 0x2228c60
  44.   -83> 2014-08-11 22:58:33.430948 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.10( empty local-les=17529 n=0 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  45.    -82> 2014-08-11 22:58:33.430960 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(2.10 epoch 17542) v2 -- ?+0 0x1ced2600 con 0x2228c60
  46.    -81> 2014-08-11 22:58:33.431007 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.17( v 17397'310111 (16278'307110,17397'310111] local-les=17529 n=4321 ec=15501 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  47.   -80> 2014-08-11 22:58:33.431021 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(6.17 epoch 17542) v2 -- ?+0 0x1ced2400 con 0x2228dc0
  48.   -79> 2014-08-11 22:58:33.431065 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.13( empty local-les=17529 n=0 ec=1 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  49.    -78> 2014-08-11 22:58:33.431078 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(2.13 epoch 17542) v2 -- ?+0 0x1ced2200 con 0x2228dc0
  50.    -77> 2014-08-11 22:58:33.431113 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.15( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  51.    -76> 2014-08-11 22:58:33.431123 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.15(7) epoch 17542) v4 -- ?+0 0x1cede540 con 0x2228dc0
  52.    -75> 2014-08-11 22:58:33.431152 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.16( v 17536'277142 (16552'274137,17536'277142] local-les=17529 n=4355 ec=15501 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17536 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  53.   -74> 2014-08-11 22:58:33.431170 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(6.16 epoch 17542) v2 -- ?+0 0x1ced2000 con 0x2228dc0
  54.   -73> 2014-08-11 22:58:33.431207 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.11( v 4702'712 (0'0,4702'712] local-les=17529 n=5 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  55.    -72> 2014-08-11 22:58:33.431230 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.11 epoch 17542) v2 -- ?+0 0x1cee9e00 con 0x2228c60
  56.    -71> 2014-08-11 22:58:33.431267 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.10( v 9147'6333 (4702'5332,9147'6333] local-les=17530 n=5 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  57.    -70> 2014-08-11 22:58:33.431279 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(0.10(7) epoch 17542) v4 -- ?+0 0x1cede380 con 0x2228dc0
  58.    -69> 2014-08-11 22:58:33.431310 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.12( empty local-les=17530 n=0 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 inactive NOTIFY] state<Start>: transitioning to Stray
  59.    -68> 2014-08-11 22:58:33.431321 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(2.12(3) epoch 17542) v4 -- ?+0 0x1cede1c0 con 0x2228c60
  60.    -67> 2014-08-11 22:58:33.431362 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.14( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  61.    -66> 2014-08-11 22:58:33.431373 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.14(7) epoch 17542) v4 -- ?+0 0x1cede000 con 0x2228dc0
  62.    -65> 2014-08-11 22:58:33.431470 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[6.f( v 17397'405072 (16403'402071,17397'405072] local-les=17529 n=4401 ec=15501 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  63.   -64> 2014-08-11 22:58:33.431490 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(6.f epoch 17542) v2 -- ?+0 0x1ced8000 con 0x2228dc0
  64.   -63> 2014-08-11 22:58:33.431535 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[1.8( v 4705'559 (0'0,4705'559] local-les=17529 n=0 ec=1 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  65.    -62> 2014-08-11 22:58:33.431553 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(1.8 epoch 17542) v2 -- ?+0 0x1ceece00 con 0x2228dc0
  66.    -61> 2014-08-11 22:58:33.431597 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[0.9( v 9147'29945 (4705'28945,9147'29945] local-les=17529 n=9 ec=1 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  67.   -60> 2014-08-11 22:58:33.431611 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(0.9 epoch 17542) v2 -- ?+0 0x1ceecc00 con 0x2228dc0
  68.   -59> 2014-08-11 22:58:33.431643 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[2.b( empty local-les=17530 n=0 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 inactive NOTIFY] state<Start>: transitioning to Stray
  69.   -58> 2014-08-11 22:58:33.431654 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(2.b(3) epoch 17542) v4 -- ?+0 0x1cedc8c0 con 0x2228c60
  70.   -57> 2014-08-11 22:58:33.431687 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[6.e( v 17461'380259 (17262'377257,17461'380259] local-les=17531 n=4358 ec=15501 les/c 17531/17534 17527/17527/17408) [21,19] r=1 lpr=17534 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  71.   -56> 2014-08-11 22:58:33.431705 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(6.e(3) epoch 17542) v4 -- ?+0 0x1cedc700 con 0x2228c60
  72.   -55> 2014-08-11 22:58:33.431745 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[1.9( v 237'46 (0'0,237'46] local-les=17529 n=5 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  73.    -54> 2014-08-11 22:58:33.431763 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.9 epoch 17542) v2 -- ?+0 0x1ceeca00 con 0x2228c60
  74.    -53> 2014-08-11 22:58:33.431805 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[4.c( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  75.    -52> 2014-08-11 22:58:33.431816 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.c(7) epoch 17542) v4 -- ?+0 0x1cedc540 con 0x2228dc0
  76.    -51> 2014-08-11 22:58:33.431869 7fab648ee700  1 osd.19 pg_epoch: 17542 pg[6.1( v 17397'194365 (16264'191364,17397'194365] local-les=17529 n=2217 ec=15501 les/c 17529/17530 17527/17527/17527) [19,20] r=0 lpr=17530 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  77.   -50> 2014-08-11 22:58:33.431883 7fab648ee700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_query(6.1 epoch 17542) v2 -- ?+0 0x1ceec800 con 0x2228dc0
  78.   -49> 2014-08-11 22:58:33.433357 7fab648ee700 -1 ./include/interval_set.h: In function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' thread 7fab648ee700 time 2014-08-11 22:58:33.432003
  79. ./include/interval_set.h: 391: FAILED assert(p->second >= before+len)
  80.  
  81. ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90)
  82. 1: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6f1f06]
  83. 2: (PGPool::update(std::tr1::shared_ptr<OSDMap const>)+0x244) [0x6d5ae4]
  84. 3: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, PG::RecoveryCtx*)+0x114) [0x6d5ce4]
  85. 4: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x206) [0x6339e6]
  86. 5: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x634054]
  87. 6: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x66f7a2]
  88. 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x83c016]
  89. 8: (ThreadPool::WorkThread::entry()+0x10) [0x83de40]
  90. 9: (()+0x7e9a) [0x7fab76114e9a]
  91. 10: (clone()+0x6d) [0x7fab74992ccd]
  92. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  93.  
  94.   -48> 2014-08-11 22:58:33.433476 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.9( v 17397'144184 (16264'142179,17397'144184] local-les=17529 n=2126 ec=15501 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  95.    -47> 2014-08-11 22:58:33.433512 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(6.9 epoch 17542) v2 -- ?+0 0x1cee9c00 con 0x2228c60
  96.    -46> 2014-08-11 22:58:33.433560 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.e( v 4671'842 (0'0,4671'842] local-les=17529 n=2 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  97.   -45> 2014-08-11 22:58:33.433581 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.e epoch 17542) v2 -- ?+0 0x1cee9a00 con 0x2228c60
  98.   -44> 2014-08-11 22:58:33.433622 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.f( v 9147'29778 (4705'28778,9147'29778] local-les=17529 n=11 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  99.    -43> 2014-08-11 22:58:33.433637 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(0.f epoch 17542) v2 -- ?+0 0x1cee9800 con 0x2228c60
  100.    -42> 2014-08-11 22:58:33.433678 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.d( empty local-les=17529 n=0 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  101.   -41> 2014-08-11 22:58:33.433692 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(2.d epoch 17542) v2 -- ?+0 0x1cee9600 con 0x2228c60
  102.   -40> 2014-08-11 22:58:33.433730 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.b( empty local-les=17529 n=0 ec=9 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  103.    -39> 2014-08-11 22:58:33.433748 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(4.b epoch 17542) v2 -- ?+0 0x1cee9400 con 0x2228c60
  104.    -38> 2014-08-11 22:58:33.433790 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.f( v 2830'1326 (150'326,2830'1326] local-les=17530 n=5 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  105.    -37> 2014-08-11 22:58:33.433809 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(1.f(7) epoch 17542) v4 -- ?+0 0x1cef8e00 con 0x2228dc0
  106.    -36> 2014-08-11 22:58:33.433842 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.c( v 4705'1052 (194'52,4705'1052] local-les=17530 n=3 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  107.    -35> 2014-08-11 22:58:33.433857 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(1.c(3) epoch 17542) v4 -- ?+0 0x1cef8c40 con 0x2228c60
  108.    -34> 2014-08-11 22:58:33.433889 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.d( v 9147'5227 (4702'4227,9147'5227] local-les=17530 n=10 ec=1 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  109.    -33> 2014-08-11 22:58:33.433901 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(0.d(3) epoch 17542) v4 -- ?+0 0x1cef8a80 con 0x2228c60
  110.    -32> 2014-08-11 22:58:33.433935 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.9( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17408) [21,19] r=1 lpr=17533 pi=17405-17526/3 inactive NOTIFY] state<Start>: transitioning to Stray
  111.    -31> 2014-08-11 22:58:33.433945 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_notify(4.9(3) epoch 17542) v4 -- ?+0 0x1cef88c0 con 0x2228c60
  112.    -30> 2014-08-11 22:58:33.433991 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.a( v 17414'167924 (16278'164921,17414'167924] local-les=17530 n=2148 ec=15501 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  113.    -29> 2014-08-11 22:58:33.434008 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(6.a(7) epoch 17542) v4 -- ?+0 0x1cef8700 con 0x2228dc0
  114.    -28> 2014-08-11 22:58:33.434050 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.e( empty local-les=17530 n=0 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  115.    -27> 2014-08-11 22:58:33.434061 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(2.e(7) epoch 17542) v4 -- ?+0 0x1cef8540 con 0x2228dc0
  116.    -26> 2014-08-11 22:58:33.434101 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.d( v 17536'345128 (16290'342125,17536'345128] local-les=17530 n=4274 ec=15501 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17536 pi=16737-17526/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  117.    -25> 2014-08-11 22:58:33.434114 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(6.d(12) epoch 17542) v4 -- ?+0 0x1cef8380 con 0x2228dc0
  118.    -24> 2014-08-11 22:58:33.434152 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.a( v 2830'657 (0'0,2830'657] local-les=17529 n=2 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  119.   -23> 2014-08-11 22:58:33.434167 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.a epoch 17542) v2 -- ?+0 0x1cee9200 con 0x2228c60
  120.   -22> 2014-08-11 22:58:33.434203 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.b( v 9147'16329 (4705'15328,9147'16329] local-les=17529 n=16 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  121.    -21> 2014-08-11 22:58:33.434218 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(0.b epoch 17542) v2 -- ?+0 0x1cee9000 con 0x2228c60
  122.    -20> 2014-08-11 22:58:33.434255 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.9( empty local-les=17529 n=0 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  123.   -19> 2014-08-11 22:58:33.434271 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(2.9 epoch 17542) v2 -- ?+0 0x1cefee00 con 0x2228c60
  124.   -18> 2014-08-11 22:58:33.434314 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.f( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16737-17526/12 inactive NOTIFY] state<Start>: transitioning to Stray
  125.   -17> 2014-08-11 22:58:33.434325 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.f(12) epoch 17542) v4 -- ?+0 0x1cef81c0 con 0x2228dc0
  126.   -16> 2014-08-11 22:58:33.434370 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.c( v 17414'559929 (17396'556925,17414'559929] local-les=17529 n=4356 ec=15501 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  127.    -15> 2014-08-11 22:58:33.434384 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(6.c epoch 17542) v2 -- ?+0 0x1cefec00 con 0x2228c60
  128.    -14> 2014-08-11 22:58:33.434424 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.a( v 9147'1893 (2830'893,9147'1893] local-les=17529 n=8 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  129.   -13> 2014-08-11 22:58:33.434438 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(0.a epoch 17542) v2 -- ?+0 0x1cefea00 con 0x2228c60
  130.   -12> 2014-08-11 22:58:33.434484 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[2.8( empty local-les=17529 n=0 ec=1 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  131.    -11> 2014-08-11 22:58:33.434497 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(2.8 epoch 17542) v2 -- ?+0 0x1cefe800 con 0x2228c60
  132.    -10> 2014-08-11 22:58:33.434534 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.e( empty local-les=17529 n=0 ec=9 les/c 17529/17532 17527/17527/17527) [19,21] r=0 lpr=17532 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  133.    -9> 2014-08-11 22:58:33.434546 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(4.e epoch 17542) v2 -- ?+0 0x1cefe600 con 0x2228c60
  134.    -8> 2014-08-11 22:58:33.434615 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[1.5( v 4702'375 (0'0,4702'375] local-les=17531 n=4 ec=1 les/c 17531/17533 17527/17527/17527) [19,21] r=0 lpr=17533 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  135.     -7> 2014-08-11 22:58:33.434630 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(1.5 epoch 17542) v2 -- ?+0 0x1cefe400 con 0x2228c60
  136.     -6> 2014-08-11 22:58:33.434666 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[0.4( v 9147'3618 (2823'2617,9147'3618] local-les=17530 n=7 ec=1 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  137.     -5> 2014-08-11 22:58:33.434678 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(0.4(7) epoch 17542) v4 -- ?+0 0x1cef8000 con 0x2228dc0
  138.     -4> 2014-08-11 22:58:33.434714 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[4.0( empty local-les=17530 n=0 ec=9 les/c 17530/17531 17527/17527/17403) [20,19] r=1 lpr=17532 pi=16739-17526/7 inactive NOTIFY] state<Start>: transitioning to Stray
  139.     -3> 2014-08-11 22:58:33.434729 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6804/3147 -- pg_notify(4.0(7) epoch 17542) v4 -- ?+0 0x1cf01e00 con 0x2228dc0
  140.     -2> 2014-08-11 22:58:33.434742 7fab650ef700  1 osd.19 pg_epoch: 17542 pg[6.5( v 17534'229610 (16290'226920,17534'229610] local-les=17541 n=2118 ec=15501 les/c 17541/17541 17527/17540/17532) [19,21] r=0 lpr=17541 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  141.    -1> 2014-08-11 22:58:33.434757 7fab650ef700  1 -- 10.100.250.1:6801/26683 --> 10.100.250.1:6807/4226 -- pg_query(6.5 epoch 17542) v2 -- ?+0 0x1cefe200 con 0x2228c60
  142.     0> 2014-08-11 22:58:33.436035 7fab650ef700 -1 ./include/interval_set.h: In function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' thread 7fab650ef700 time 2014-08-11 22:58:33.434841
  143. ./include/interval_set.h: 391: FAILED assert(p->second >= before+len)
  144.  
  145. ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90)
  146. 1: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6f1f06]
  147. 2: (PGPool::update(std::tr1::shared_ptr<OSDMap const>)+0x244) [0x6d5ae4]
  148. 3: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, PG::RecoveryCtx*)+0x114) [0x6d5ce4]
  149. 4: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x206) [0x6339e6]
  150. 5: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x634054]
  151. 6: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x66f7a2]
  152. 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x83c016]
  153. 8: (ThreadPool::WorkThread::entry()+0x10) [0x83de40]
  154. 9: (()+0x7e9a) [0x7fab76114e9a]
  155. 10: (clone()+0x6d) [0x7fab74992ccd]
  156. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  157.  
  158. --- logging levels ---
  159.   0/ 5 none
  160.   0/ 1 lockdep
  161.   0/ 1 context
  162.   1/ 1 crush
  163.   1/ 5 mds
  164.   1/ 5 mds_balancer
  165.   1/ 5 mds_locker
  166.   1/ 5 mds_log
  167.   1/ 5 mds_log_expire
  168.   1/ 5 mds_migrator
  169.   0/ 1 buffer
  170.   0/ 1 timer
  171.   0/ 1 filer
  172.   0/ 1 striper
  173.   0/ 1 objecter
  174.   0/ 5 rados
  175.   0/ 5 rbd
  176.   0/ 5 journaler
  177.   0/ 5 objectcacher
  178.   0/ 5 client
  179.   3/ 5 osd
  180.   0/ 5 optracker
  181.   0/ 5 objclass
  182.   1/ 3 filestore
  183.   1/ 3 journal
  184.   0/ 5 ms
  185.   1/ 5 mon
  186.   0/10 monc
  187.   0/ 5 paxos
  188.   0/ 5 tp
  189.   1/ 5 auth
  190.   1/ 5 crypto
  191.   1/ 1 finisher
  192.   1/ 5 heartbeatmap
  193.   1/ 5 perfcounter
  194.   1/ 5 rgw
  195.   1/ 5 hadoop
  196.   1/ 5 javaclient
  197.   1/ 5 asok
  198.   1/ 1 throttle
  199.  -2/-2 (syslog threshold)
  200.  -1/-1 (stderr threshold)
  201.  max_recent     10000
  202.  max_new         1000
  203.  log_file /var/log/ceph/ceph-osd.19.log
  204. --- end dump of recent events ---
  205. --- begin dump of recent events ---
  206. --- logging levels ---
  207.   0/ 5 none
  208.   0/ 1 lockdep
  209.   0/ 1 context
  210.   1/ 1 crush
  211.   1/ 5 mds
  212.   1/ 5 mds_balancer
  213.   1/ 5 mds_locker
  214.   1/ 5 mds_log
  215.   1/ 5 mds_log_expire
  216.   1/ 5 mds_migrator
  217.   0/ 1 buffer
  218.   0/ 1 timer
  219.   0/ 1 filer
  220.   0/ 1 striper
  221.   0/ 1 objecter
  222.   0/ 5 rados
  223.   0/ 5 rbd
  224.   0/ 5 journaler
  225.   0/ 5 objectcacher
  226.   0/ 5 client
  227.   3/ 5 osd
  228.   0/ 5 optracker
  229.   0/ 5 objclass
  230.   1/ 3 filestore
  231.   1/ 3 journal
  232.   0/ 5 ms
  233.   1/ 5 mon
  234.   0/10 monc
  235.   0/ 5 paxos
  236.   0/ 5 tp
  237.   1/ 5 auth
  238.   1/ 5 crypto
  239.   1/ 1 finisher
  240.   1/ 5 heartbeatmap
  241.   1/ 5 perfcounter
  242.   1/ 5 rgw
  243.   1/ 5 hadoop
  244.   1/ 5 javaclient
  245.   1/ 5 asok
  246.   1/ 1 throttle
  247.  -2/-2 (syslog threshold)
  248.  -1/-1 (stderr threshold)
  249.  max_recent     10000
  250.  max_new         1000
  251.  log_file /var/log/ceph/ceph-osd.19.log
  252. --- end dump of recent events ---
  253. 2014-08-11 22:58:33.492016 7fab648ee700 -1 *** Caught signal (Aborted) **
  254. in thread 7fab648ee700
  255.  
  256. ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90)
  257. 1: /usr/bin/ceph-osd() [0x79582a]
  258. 2: (()+0xfcb0) [0x7fab7611ccb0]
  259. 3: (gsignal()+0x35) [0x7fab748d5425]
  260. 4: (abort()+0x17b) [0x7fab748d8b8b]
  261. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab7522769d]
  262. 6: (()+0xb5846) [0x7fab75225846]
  263. 7: (()+0xb5873) [0x7fab75225873]
  264. 8: (()+0xb596e) [0x7fab7522596e]
  265. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x846bdf]
  266. 10: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6f1f06]
  267. 11: (PGPool::update(std::tr1::shared_ptr<OSDMap const>)+0x244) [0x6d5ae4]
  268. 12: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, PG::RecoveryCtx*)+0x114) [0x6d5ce4]
  269. 13: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x206) [0x6339e6]
  270. 14: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x634054]
  271. 15: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x66f7a2]
  272. 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x83c016]
  273. 17: (ThreadPool::WorkThread::entry()+0x10) [0x83de40]
  274. 18: (()+0x7e9a) [0x7fab76114e9a]
  275. 19: (clone()+0x6d) [0x7fab74992ccd]
  276. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  277.  
  278. --- begin dump of recent events ---
  279.     0> 2014-08-11 22:58:33.492016 7fab648ee700 -1 *** Caught signal (Aborted) **
  280. in thread 7fab648ee700
  281.  
  282. ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90)
  283. 1: /usr/bin/ceph-osd() [0x79582a]
  284. 2: (()+0xfcb0) [0x7fab7611ccb0]
  285. 3: (gsignal()+0x35) [0x7fab748d5425]
  286. 4: (abort()+0x17b) [0x7fab748d8b8b]
  287. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab7522769d]
  288. 6: (()+0xb5846) [0x7fab75225846]
  289. 7: (()+0xb5873) [0x7fab75225873]
  290. 8: (()+0xb596e) [0x7fab7522596e]
  291. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x846bdf]
  292. 10: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6f1f06]
  293. 11: (PGPool::update(std::tr1::shared_ptr<OSDMap const>)+0x244) [0x6d5ae4]
  294. 12: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, PG::RecoveryCtx*)+0x114) [0x6d5ce4]
  295. 13: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x206) [0x6339e6]
  296. 14: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x634054]
  297. 15: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x66f7a2]
  298. 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x83c016]
  299. 17: (ThreadPool::WorkThread::entry()+0x10) [0x83de40]
  300. 18: (()+0x7e9a) [0x7fab76114e9a]
  301. 19: (clone()+0x6d) [0x7fab74992ccd]
  302. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  303.  
  304. --- logging levels ---
  305.   0/ 5 none
  306.   0/ 1 lockdep
  307.   0/ 1 context
  308.   1/ 1 crush
  309.   1/ 5 mds
  310.   1/ 5 mds_balancer
  311.   1/ 5 mds_locker
  312.   1/ 5 mds_log
  313.   1/ 5 mds_log_expire
  314.   1/ 5 mds_migrator
  315.   0/ 1 buffer
  316.   0/ 1 timer
  317.   0/ 1 filer
  318.   0/ 1 striper
  319.   0/ 1 objecter
  320.   0/ 5 rados
  321.   0/ 5 rbd
  322.   0/ 5 journaler
  323.   0/ 5 objectcacher
  324.   0/ 5 client
  325.   3/ 5 osd
  326.   0/ 5 optracker
  327.   0/ 5 objclass
  328.   1/ 3 filestore
  329.   1/ 3 journal
  330.   0/ 5 ms
  331.   1/ 5 mon
  332.   0/10 monc
  333.   0/ 5 paxos
  334.   0/ 5 tp
  335.   1/ 5 auth
  336.   1/ 5 crypto
  337.   1/ 1 finisher
  338.   1/ 5 heartbeatmap
  339.   1/ 5 perfcounter
  340.   1/ 5 rgw
  341.   1/ 5 hadoop
  342.   1/ 5 javaclient
  343.   1/ 5 asok
  344.   1/ 1 throttle
  345.  -2/-2 (syslog threshold)
  346.  -1/-1 (stderr threshold)
  347.  max_recent     10000
  348.  max_new         1000
  349.  log_file /var/log/ceph/ceph-osd.19.log
  350. --- end dump of recent events ---
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!