API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Aug 15th, 2014
38
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 39.76 KB | None | 0 0
raw download clone embed print report
  1. -142> 2014-08-15 14:43:49.465571 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 807 ==== pg_info(1 pgs e18841:1.10a) v3 ==== 588+0+0 (1484367401 0 0) 0x1d2a8000 con 0x34bbdc0
  2. -141> 2014-08-15 14:43:49.465584 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.464736, event: header_read, request: pg_info(1 pgs e18841:1.10a) v3
  3. -140> 2014-08-15 14:43:49.465858 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.4b epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e719100
  4. -139> 2014-08-15 14:43:49.465742 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.464739, event: throttled, request: pg_info(1 pgs e18841:1.10a) v3
  5. -138> 2014-08-15 14:43:49.465927 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.464867, event: all_read, request: pg_info(1 pgs e18841:1.10a) v3
  6. -137> 2014-08-15 14:43:49.465932 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.465583, event: dispatched, request: pg_info(1 pgs e18841:1.10a) v3
  7. -136> 2014-08-15 14:43:49.465936 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.465936, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:1.10a) v3
  8. -135> 2014-08-15 14:43:49.465949 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.465949, event: started, request: pg_info(1 pgs e18841:1.10a) v3
  9. -134> 2014-08-15 14:43:49.465963 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 812, time: 2014-08-15 14:43:49.465963, event: done, request: pg_info(1 pgs e18841:1.10a) v3
  10. -133> 2014-08-15 14:43:49.465989 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 808 ==== pg_info(1 pgs e18841:0.10b) v3 ==== 588+0+0 (466741130 0 0) 0x1d2a8c40 con 0x34bbdc0
  11. -132> 2014-08-15 14:43:49.466000 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.464880, event: header_read, request: pg_info(1 pgs e18841:0.10b) v3
  12. -131> 2014-08-15 14:43:49.466004 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.464881, event: throttled, request: pg_info(1 pgs e18841:0.10b) v3
  13. -130> 2014-08-15 14:43:49.466012 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.465183, event: all_read, request: pg_info(1 pgs e18841:0.10b) v3
  14. -129> 2014-08-15 14:43:49.466015 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.465999, event: dispatched, request: pg_info(1 pgs e18841:0.10b) v3
  15. -128> 2014-08-15 14:43:49.466018 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.466018, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:0.10b) v3
  16. -127> 2014-08-15 14:43:49.466031 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.466031, event: started, request: pg_info(1 pgs e18841:0.10b) v3
  17. -126> 2014-08-15 14:43:49.466041 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 813, time: 2014-08-15 14:43:49.466041, event: done, request: pg_info(1 pgs e18841:0.10b) v3
  18. -125> 2014-08-15 14:43:49.466048 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 809 ==== MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1 ==== 25+0+0 (1029305122 0 0) 0x344d700 con 0x34bbdc0
  19. -124> 2014-08-15 14:43:49.466058 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.465287, event: header_read, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  20. -123> 2014-08-15 14:43:49.466061 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.465300, event: throttled, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  21. -122> 2014-08-15 14:43:49.466070 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.465388, event: all_read, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  22. -121> 2014-08-15 14:43:49.466073 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.466057, event: dispatched, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  23. -120> 2014-08-15 14:43:49.466076 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.466076, event: waiting_for_osdmap, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  24. -119> 2014-08-15 14:43:49.466092 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 814, time: 2014-08-15 14:43:49.466091, event: done, request: MRecoveryReserve REQUEST pgid: 6.38, query_epoch: 18841 v1
  25. -118> 2014-08-15 14:43:49.466099 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 810 ==== MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1 ==== 25+0+0 (1631588381 0 0) 0x1cf33540 con 0x34bbdc0
  26. -117> 2014-08-15 14:43:49.466104 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.465410, event: header_read, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  27. -116> 2014-08-15 14:43:49.466107 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.465411, event: throttled, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  28. -115> 2014-08-15 14:43:49.466110 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.465477, event: all_read, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  29. -114> 2014-08-15 14:43:49.466113 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.466104, event: dispatched, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  30. -113> 2014-08-15 14:43:49.466116 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.466116, event: waiting_for_osdmap, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  31. -112> 2014-08-15 14:43:49.466122 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 815, time: 2014-08-15 14:43:49.466122, event: done, request: MRecoveryReserve REQUEST pgid: 6.37, query_epoch: 18841 v1
  32. -111> 2014-08-15 14:43:49.465723 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.5c( v 9147'14099 (4705'13099,9147'14099] local-les=18837 n=11 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  33. -110> 2014-08-15 14:43:49.466004 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(4.5e(31),4.5d(31),1.56(31),0.57(31),2.55(31),4.53(31) epoch 18841) v4 -- ?+0 0x1d2c0700 con 0x34bbdc0
  34. -109> 2014-08-15 14:43:49.466229 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(6 pgs e18841:0.52(30),1.51(30),6.49(30),1.4e(30),0.4f(30),2.4d(30)) v3 -- ?+0 0x1d2c08c0 con 0x34bbdc0
  35. -108> 2014-08-15 14:43:49.466461 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.58 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e6be600
  36. -107> 2014-08-15 14:43:49.466659 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(2.5f(31) epoch 18841) v4 -- ?+0 0x1d84be00 con 0x34bbdc0
  37. -106> 2014-08-15 14:43:49.466710 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(6 pgs e18841:0.5e(30),1.5c(30),0.5d(30),1.5d(30),0.5c(30),1.5b(30)) v3 -- ?+0 0x1d84bc40 con 0x34bbdc0
  38. -105> 2014-08-15 14:43:49.466927 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.4e epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e6be080
  39. -104> 2014-08-15 14:43:49.467420 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.43 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e719680
  40. -103> 2014-08-15 14:43:49.467499 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.49( v 15734'8854 (4705'7853,15734'8854] local-les=18837 n=12 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  41. -102> 2014-08-15 14:43:49.468002 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.45( v 9147'6512 (4702'5511,9147'6512] local-les=18837 n=14 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  42. -101> 2014-08-15 14:43:49.468538 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.47( v 9147'12420 (4702'11419,9147'12420] local-les=18837 n=7 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  43. -100> 2014-08-15 14:43:49.468569 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.41 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db76000
  44. -99> 2014-08-15 14:43:49.469021 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.45 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e6bdb00
  45. -98> 2014-08-15 14:43:49.469095 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(1.4c(31),0.4d(31),4.49(31),2.4b(31) epoch 18841) v4 -- ?+0 0x1d7718c0 con 0x34bbdc0
  46. -97> 2014-08-15 14:43:49.469127 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(5 pgs e18841:1.48(30),0.49(30),6.41(30),1.46(30),0.47(30)) v3 -- ?+0 0x1d771700 con 0x34bbdc0
  47. -96> 2014-08-15 14:43:49.469184 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.45 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e6bd580
  48. -95> 2014-08-15 14:43:49.469663 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.3f epoch 18841 query_epoch 18841) v3 -- ?+0 0x1e6bd000
  49. -94> 2014-08-15 14:43:49.469792 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(6.38(31),0.3e(31),2.3c(31),4.3a(31),1.3c(31),0.3d(31),4.39(31),1.3d(31) epoch 18841) v4 -- ?+0 0x1d771540 con 0x34bbdc0
  50. -93> 2014-08-15 14:43:49.469809 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(1 pgs e18841:6.3b(30)) v3 -- ?+0 0x1d771380 con 0x34bbdc0
  51. -92> 2014-08-15 14:43:49.469834 7fa82990e700 5 --OSD::tracker-- reqid: osd.20.0:6, seq: 643, time: 2014-08-15 14:43:49.469833, event: reached_pg, request: osd_sub_op(osd.20.0:6 6.38 a9abf8/rbd_data.d81262ae8944a.0000000000000302/head//6 [push] v 18738'302382 snapset=0=[]:[] snapc=0=[]) v7
  52. -91> 2014-08-15 14:43:49.470241 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.3d epoch 18841 query_epoch 18841) v3 -- ?+0 0x1da54080
  53. -90> 2014-08-15 14:43:49.470270 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.41( v 9147'1671 (237'671,9147'1671] local-les=18837 n=15 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  54. -89> 2014-08-15 14:43:49.470399 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.43 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db76580
  55. -88> 2014-08-15 14:43:49.470742 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(6.40(31),0.46(31),2.44(31),4.42(31),1.45(31),6.47(31) epoch 18841) v4 -- ?+0 0x1d0a4a80 con 0x34bbdc0
  56. -87> 2014-08-15 14:43:49.471460 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(1.38(31),0.39(31),2.3b(31),6.33(31),2.37(31) epoch 18841) v4 -- ?+0 0x1d771000 con 0x34bbdc0
  57. -86> 2014-08-15 14:43:49.471519 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(1 pgs e18841:6.3f(30)) v3 -- ?+0 0x1dbd2e00 con 0x34bbdc0
  58. -85> 2014-08-15 14:43:49.471558 7fa82990e700 5 --OSD::tracker-- reqid: osd.20.0:7, seq: 644, time: 2014-08-15 14:43:49.471557, event: reached_pg, request: osd_sub_op(osd.20.0:7 6.38 899ab38/rbd_data.110f7e2ae8944a.0000000000001c2d/head//6 [push] v 18738'302381 snapset=0=[]:[] snapc=0=[]) v7
  59. -84> 2014-08-15 14:43:49.471818 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(6.37(31),4.35(31),6.28(31) epoch 18841) v4 -- ?+0 0x1dbd2c40 con 0x34bbdc0
  60. -83> 2014-08-15 14:43:49.471835 7fa82990e700 5 --OSD::tracker-- reqid: osd.20.0:8, seq: 645, time: 2014-08-15 14:43:49.471835, event: reached_pg, request: osd_sub_op(osd.20.0:8 6.37 6e9433f7/rbd_data.726a62ae8944a.0000000000000024/head//6 [push] v 18738'414497 snapset=0=[]:[] snapc=0=[]) v7
  61. -82> 2014-08-15 14:43:49.471988 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.2c( v 9147'25226 (4702'24226,9147'25226] local-les=18837 n=18 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  62. -81> 2014-08-15 14:43:49.471473 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(6 pgs e18841:6.43(30),1.44(30),0.45(30),2.47(30),1.40(30),0.41(30)) v3 -- ?+0 0x1d0a4380 con 0x34bbdc0
  63. -80> 2014-08-15 14:43:49.472167 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.24( v 9147'18781 (4705'17780,9147'18781] local-les=18837 n=5 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  64. -79> 2014-08-15 14:43:49.472391 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.20 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db76b00
  65. -78> 2014-08-15 14:43:49.472431 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.28 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db09580
  66. -77> 2014-08-15 14:43:49.472986 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.2a epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db09b00
  67. -76> 2014-08-15 14:43:49.473312 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.25( v 9147'5191 (3041'4190,9147'5191] local-les=18837 n=8 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  68. -75> 2014-08-15 14:43:49.473456 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.21 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db0a080
  69. -74> 2014-08-15 14:43:49.473521 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(0.2e(31),2.2c(31),4.2a(31),1.2d(31) epoch 18841) v4 -- ?+0 0x1dbd2700 con 0x34bbdc0
  70. -73> 2014-08-15 14:43:49.473551 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(4 pgs e18841:0.2c(30),1.2b(30),1.24(30),0.25(30)) v3 -- ?+0 0x1dbd2540 con 0x34bbdc0
  71. -72> 2014-08-15 14:43:49.473659 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.1a( v 9147'2784 (4666'1783,9147'2784] local-les=18837 n=12 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  72. -71> 2014-08-15 14:43:49.474779 7fa82a10f700 2 filestore(/var/lib/ceph/osd/ceph01-21) waiting 51 > 50 ops || 96685 > 104857600
  73. -70> 2014-08-15 14:43:49.474803 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(6.1c(31),4.1e(31),4.1d(31) epoch 18841) v4 -- ?+0 0x1dbd21c0 con 0x34bbdc0
  74. -69> 2014-08-15 14:43:49.474867 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(5 pgs e18841:0.1a(30),2.18(30),6.1f(30),6.1e(30),1.19(30)) v3 -- ?+0 0x1dbd2000 con 0x34bbdc0
  75. -68> 2014-08-15 14:43:49.475235 7fa82990e700 2 filestore(/var/lib/ceph/osd/ceph01-21) waiting 51 > 50 ops || 96749 > 104857600
  76. -67> 2014-08-15 14:43:49.478521 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.23 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db77080
  77. -66> 2014-08-15 14:43:49.478725 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.16 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db0a600
  78. -65> 2014-08-15 14:43:49.479231 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.22 epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db77600
  79. -64> 2014-08-15 14:43:49.479894 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(1.10(31),0.11(31) epoch 18841) v4 -- ?+0 0x1d771a80 con 0x34bbdc0
  80. -63> 2014-08-15 14:43:49.479986 7fa82990e700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(2 pgs e18841:6.14(30),6.8(30)) v3 -- ?+0 0x1d771c40 con 0x34bbdc0
  81. -62> 2014-08-15 14:43:49.480233 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.1f( v 9147'2311 (247'1310,9147'2311] local-les=18837 n=12 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  82. -61> 2014-08-15 14:43:49.480295 7fa82990e700 5 --OSD::tracker-- reqid: client.1114846.0:2230, seq: 450, time: 2014-08-15 14:43:49.480294, event: reached_pg, request: osd_op(client.1114846.0:2230 rbd_header.788ce2ae8944a [watch add cookie 1 ver 0] 6.8baeb4d4 e18841) v4
  83. -60> 2014-08-15 14:43:49.480579 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.1b epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db77b80
  84. -59> 2014-08-15 14:43:49.480788 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(0.22(31),2.20(31),1.20(31),0.21(31),1.21(31),6.1b(31),2.1f(31) epoch 18841) v4 -- ?+0 0x1dcb9e00 con 0x34bbdc0
  85. -58> 2014-08-15 14:43:49.480856 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(7 pgs e18841:0.24(30),1.23(30),6.26(30),6.19(30),1.1e(30),0.1f(30),2.1d(30)) v3 -- ?+0 0x1dcb91c0 con 0x34bbdc0
  86. -57> 2014-08-15 14:43:49.481191 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.e( v 15734'3771 (245'2771,15734'3771] local-les=18837 n=13 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  87. -56> 2014-08-15 14:43:49.481374 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(2.c epoch 18841 query_epoch 18841) v3 -- ?+0 0x1db80000
  88. -55> 2014-08-15 14:43:49.481460 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.a epoch 18841 query_epoch 18841) v3 -- ?+0 0x34dc100
  89. -54> 2014-08-15 14:43:49.481708 7fa82a10f700 3 osd.21 pg_epoch: 18841 pg[0.c( v 9147'8997 (4650'7996,9147'8997] local-les=18837 n=9 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  90. -53> 2014-08-15 14:43:49.481861 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> osd.20 10.100.250.1:6801/7231 -- pg_log(4.8 epoch 18841 query_epoch 18841) v3 -- ?+0 0x34db600
  91. -52> 2014-08-15 14:43:49.482059 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_notify(6.b(31),2.f(31) epoch 18841) v4 -- ?+0 0x1dcb9700 con 0x34bbdc0
  92. -51> 2014-08-15 14:43:49.482085 7fa82a10f700 1 -- 10.100.250.1:6805/12417 --> 10.100.250.1:6801/7231 -- pg_info(4 pgs e18841:0.e(30),1.d(30),0.c(30),1.b(30)) v3 -- ?+0 0x1dc81c40 con 0x34bbdc0
  93. -50> 2014-08-15 14:43:49.495372 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 811 ==== pg_info(1 pgs e18841:2.109) v3 ==== 588+0+0 (589605993 0 0) 0x1dc81c40 con 0x34bbdc0
  94. -49> 2014-08-15 14:43:49.495444 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495095, event: header_read, request: pg_info(1 pgs e18841:2.109) v3
  95. -48> 2014-08-15 14:43:49.495456 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495103, event: throttled, request: pg_info(1 pgs e18841:2.109) v3
  96. -47> 2014-08-15 14:43:49.495461 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495280, event: all_read, request: pg_info(1 pgs e18841:2.109) v3
  97. -46> 2014-08-15 14:43:49.495466 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495434, event: dispatched, request: pg_info(1 pgs e18841:2.109) v3
  98. -45> 2014-08-15 14:43:49.495474 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495474, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:2.109) v3
  99. -44> 2014-08-15 14:43:49.495485 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495485, event: started, request: pg_info(1 pgs e18841:2.109) v3
  100. -43> 2014-08-15 14:43:49.495531 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 816, time: 2014-08-15 14:43:49.495531, event: done, request: pg_info(1 pgs e18841:2.109) v3
  101. -42> 2014-08-15 14:43:49.496122 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 812 ==== pg_info(1 pgs e18841:1.116) v3 ==== 588+0+0 (4130120021 0 0) 0x1dcb9700 con 0x34bbdc0
  102. -41> 2014-08-15 14:43:49.496224 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.495341, event: header_read, request: pg_info(1 pgs e18841:1.116) v3
  103. -40> 2014-08-15 14:43:49.496261 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.495343, event: throttled, request: pg_info(1 pgs e18841:1.116) v3
  104. -39> 2014-08-15 14:43:49.496281 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.496047, event: all_read, request: pg_info(1 pgs e18841:1.116) v3
  105. -38> 2014-08-15 14:43:49.496300 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.496207, event: dispatched, request: pg_info(1 pgs e18841:1.116) v3
  106. -37> 2014-08-15 14:43:49.496321 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.496320, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:1.116) v3
  107. -36> 2014-08-15 14:43:49.496341 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.496340, event: started, request: pg_info(1 pgs e18841:1.116) v3
  108. -35> 2014-08-15 14:43:49.496401 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 817, time: 2014-08-15 14:43:49.496401, event: done, request: pg_info(1 pgs e18841:1.116) v3
  109. -34> 2014-08-15 14:43:49.496447 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 813 ==== pg_info(1 pgs e18841:0.117) v3 ==== 588+0+0 (379514131 0 0) 0x1dcb91c0 con 0x34bbdc0
  110. -33> 2014-08-15 14:43:49.496478 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496084, event: header_read, request: pg_info(1 pgs e18841:0.117) v3
  111. -32> 2014-08-15 14:43:49.496493 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496087, event: throttled, request: pg_info(1 pgs e18841:0.117) v3
  112. -31> 2014-08-15 14:43:49.496512 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496208, event: all_read, request: pg_info(1 pgs e18841:0.117) v3
  113. -30> 2014-08-15 14:43:49.496537 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496475, event: dispatched, request: pg_info(1 pgs e18841:0.117) v3
  114. -29> 2014-08-15 14:43:49.496555 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496554, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:0.117) v3
  115. -28> 2014-08-15 14:43:49.496575 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496575, event: started, request: pg_info(1 pgs e18841:0.117) v3
  116. -27> 2014-08-15 14:43:49.496616 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 818, time: 2014-08-15 14:43:49.496616, event: done, request: pg_info(1 pgs e18841:0.117) v3
  117. -26> 2014-08-15 14:43:49.496697 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 814 ==== pg_info(1 pgs e18841:2.12b) v3 ==== 588+0+0 (4213227868 0 0) 0x1dcb9e00 con 0x34bbdc0
  118. -25> 2014-08-15 14:43:49.496728 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496233, event: header_read, request: pg_info(1 pgs e18841:2.12b) v3
  119. -24> 2014-08-15 14:43:49.496747 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496235, event: throttled, request: pg_info(1 pgs e18841:2.12b) v3
  120. -23> 2014-08-15 14:43:49.496765 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496311, event: all_read, request: pg_info(1 pgs e18841:2.12b) v3
  121. -22> 2014-08-15 14:43:49.496783 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496725, event: dispatched, request: pg_info(1 pgs e18841:2.12b) v3
  122. -21> 2014-08-15 14:43:49.496801 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496800, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:2.12b) v3
  123. -20> 2014-08-15 14:43:49.496821 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496820, event: started, request: pg_info(1 pgs e18841:2.12b) v3
  124. -19> 2014-08-15 14:43:49.496861 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 819, time: 2014-08-15 14:43:49.496861, event: done, request: pg_info(1 pgs e18841:2.12b) v3
  125. -18> 2014-08-15 14:43:49.496888 7fa82d916700 1 -- 10.100.250.1:6805/12417 <== osd.20 10.100.250.1:6801/7231 815 ==== pg_info(1 pgs e18841:0.114) v3 ==== 588+0+0 (2174020110 0 0) 0x1d771c40 con 0x34bbdc0
  126. -17> 2014-08-15 14:43:49.496914 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.496350, event: header_read, request: pg_info(1 pgs e18841:0.114) v3
  127. -16> 2014-08-15 14:43:49.496931 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.496352, event: throttled, request: pg_info(1 pgs e18841:0.114) v3
  128. -15> 2014-08-15 14:43:49.496950 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.496425, event: all_read, request: pg_info(1 pgs e18841:0.114) v3
  129. -14> 2014-08-15 14:43:49.496966 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.496912, event: dispatched, request: pg_info(1 pgs e18841:0.114) v3
  130. -13> 2014-08-15 14:43:49.496983 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.496982, event: waiting_for_osdmap, request: pg_info(1 pgs e18841:0.114) v3
  131. -12> 2014-08-15 14:43:49.497008 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.497008, event: started, request: pg_info(1 pgs e18841:0.114) v3
  132. -11> 2014-08-15 14:43:49.497046 7fa82d916700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 820, time: 2014-08-15 14:43:49.497045, event: done, request: pg_info(1 pgs e18841:0.114) v3
  133. -10> 2014-08-15 14:43:49.497805 7fa82990e700 5 --OSD::tracker-- reqid: client.1114846.0:2230, seq: 450, time: 2014-08-15 14:43:49.497804, event: started, request: osd_op(client.1114846.0:2230 rbd_header.788ce2ae8944a [watch add cookie 1 ver 0] 6.8baeb4d4 e18841) v4
  134. -9> 2014-08-15 14:43:49.497843 7fa82990e700 5 --OSD::tracker-- reqid: client.1114846.0:2230, seq: 450, time: 2014-08-15 14:43:49.497843, event: started, request: osd_op(client.1114846.0:2230 rbd_header.788ce2ae8944a [watch add cookie 1 ver 0] 6.8baeb4d4 e18841) v4
  135. -8> 2014-08-15 14:43:49.497876 7fa82990e700 1 -- 10.100.250.1:6804/12417 --> 10.100.100.4:0/1051460 -- osd_op_reply(2230 rbd_header.788ce2ae8944a [watch add cookie 1 ver 0] ondisk = 0) v4 -- ?+0 0x1dd11200 con 0x1d2fb580
  136. -7> 2014-08-15 14:43:49.497902 7fa82990e700 5 --OSD::tracker-- reqid: client.1114846.0:2230, seq: 450, time: 2014-08-15 14:43:49.497902, event: done, request: osd_op(client.1114846.0:2230 rbd_header.788ce2ae8944a [watch add cookie 1 ver 0] 6.8baeb4d4 e18841) v4
  137. -6> 2014-08-15 14:43:49.498071 7fa82990e700 3 osd.21 pg_epoch: 18841 pg[0.8( v 15734'32518 (4705'31508,15734'32518] local-les=18837 n=12 ec=1 les/c 18725/18725 18840/18840/18840) [21,20] r=0 lpr=18840 pi=18724-18839/30 lcod 0'0 mlcod 0'0 inactive] may_need_replay no known survivors of interval 18802-18803, may need replay
  138. -5> 2014-08-15 14:43:49.498131 7fa82a10f700 5 --OSD::tracker-- reqid: client.1115989.0:2197, seq: 5, time: 2014-08-15 14:43:49.498131, event: reached_pg, request: osd_op(client.1115989.0:2197 rbd_header.727e32ae8944a [watch add cookie 1 ver 0] 6.742073d4 e18840) v4
  139. -4> 2014-08-15 14:43:49.499306 7fa82a10f700 5 --OSD::tracker-- reqid: client.1115989.0:2197, seq: 5, time: 2014-08-15 14:43:49.499306, event: started, request: osd_op(client.1115989.0:2197 rbd_header.727e32ae8944a [watch add cookie 1 ver 0] 6.742073d4 e18840) v4
  140. -3> 2014-08-15 14:43:49.499338 7fa82a10f700 5 --OSD::tracker-- reqid: client.1115989.0:2197, seq: 5, time: 2014-08-15 14:43:49.499338, event: started, request: osd_op(client.1115989.0:2197 rbd_header.727e32ae8944a [watch add cookie 1 ver 0] 6.742073d4 e18840) v4
  141. -2> 2014-08-15 14:43:49.499372 7fa82a10f700 1 -- 10.100.250.1:6804/12417 --> 10.100.100.4:0/1072916 -- osd_op_reply(2197 rbd_header.727e32ae8944a [watch add cookie 1 ver 0] ondisk = 0) v4 -- ?+0 0x1d0bc600 con 0x1d2fb160
  142. -1> 2014-08-15 14:43:49.499409 7fa82a10f700 5 --OSD::tracker-- reqid: client.1115989.0:2197, seq: 5, time: 2014-08-15 14:43:49.499409, event: done, request: osd_op(client.1115989.0:2197 rbd_header.727e32ae8944a [watch add cookie 1 ver 0] 6.742073d4 e18840) v4
  143. 0> 2014-08-15 14:43:49.500560 7fa82990e700 -1 ./include/interval_set.h: In function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' thread 7fa82990e700 time 2014-08-15 14:43:49.498369
  144. ./include/interval_set.h: 391: FAILED assert(p->second >= before+len)
  145.  
  146. ceph version 0.61.9-12-ga5d0178 (a5d0178088973fdc70df692b0d8740aba40164db)
  147. 1: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6e7ba6]
  148. 2: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<pg_t, pg_query_t, std::less<pg_t>, std::allocator<std::pair<pg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<pg_t, pg_query_t, std::less<pg_t>, std::allocator<std::pair<pg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > >, std::allocator<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > > > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > >, std::allocator<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > > > > > > >*)+0x24b) [0x6c76db]
  149. 3: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x317) [0x6ca987]
  150. 4: (boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<PG::RecoveryState::Primary> const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x5c) [0x6f5e8c]
  151. 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x90) [0x6f6900]
  152. 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x182) [0x6f6b92]
  153. 7: (boost::statechart::simple_state<PG::RecoveryState::WaitFlushedPeering, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x88) [0x6f0d98]
  154. 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0xfb) [0x6db65b]
  155. 9: (PG::RecoveryState::handle_event(boost::statechart::event_base const&, PG::RecoveryCtx*)+0x57) [0x6db8d7]
  156. 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0xee) [0x69356e]
  157. 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x49d) [0x62a9cd]
  158. 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x62ada4]
  159. 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x666122]
  160. 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x833826]
  161. 15: (ThreadPool::WorkThread::entry()+0x10) [0x835650]
  162. 16: (()+0x7e9a) [0x7fa84ae77e9a]
  163. 17: (clone()+0x6d) [0x7fa84916873d]
  164. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  165.  
  166. --- logging levels ---
  167. 0/ 5 none
  168. 0/ 1 lockdep
  169. 0/ 1 context
  170. 1/ 1 crush
  171. 1/ 5 mds
  172. 1/ 5 mds_balancer
  173. 1/ 5 mds_locker
  174. 1/ 5 mds_log
  175. 1/ 5 mds_log_expire
  176. 1/ 5 mds_migrator
  177. 0/ 1 buffer
  178. 0/ 1 timer
  179. 0/ 1 filer
  180. 0/ 1 striper
  181. 0/ 1 objecter
  182. 0/ 5 rados
  183. 0/ 5 rbd
  184. 0/ 5 journaler
  185. 0/ 5 objectcacher
  186. 0/ 5 client
  187. 3/ 5 osd
  188. 0/ 5 optracker
  189. 0/ 5 objclass
  190. 1/ 3 filestore
  191. 1/ 3 journal
  192. 0/ 5 ms
  193. 1/ 5 mon
  194. 0/10 monc
  195. 0/ 5 paxos
  196. 0/ 5 tp
  197. 1/ 5 auth
  198. 1/ 5 crypto
  199. 1/ 1 finisher
  200. 1/ 5 heartbeatmap
  201. 1/ 5 perfcounter
  202. 1/ 5 rgw
  203. 1/ 5 hadoop
  204. 1/ 5 javaclient
  205. 1/ 5 asok
  206. 1/ 1 throttle
  207. -2/-2 (syslog threshold)
  208. -1/-1 (stderr threshold)
  209. max_recent 10000
  210. max_new 1000
  211. log_file /var/log/ceph/ceph-osd.21.log
  212. --- end dump of recent events ---
  213. 2014-08-15 14:43:49.504264 7fa82a10f700 -1 ./include/interval_set.h: In function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' thread 7fa82a10f700 time 2014-08-15 14:43:49.499643
  214. ./include/interval_set.h: 391: FAILED assert(p->second >= before+len)
  215.  
  216. ceph version 0.61.9-12-ga5d0178 (a5d0178088973fdc70df692b0d8740aba40164db)
  217. 1: (interval_set<snapid_t>::subtract(interval_set<snapid_t> const&)+0x206) [0x6e7ba6]
  218. 2: (PG::activate(ObjectStore::Transaction&, unsigned int, std::list<Context*, std::allocator<Context*> >&, std::map<int, std::map<pg_t, pg_query_t, std::less<pg_t>, std::allocator<std::pair<pg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<pg_t, pg_query_t, std::less<pg_t>, std::allocator<std::pair<pg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > >, std::allocator<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > > > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > >, std::allocator<std::pair<pg_notify_t, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > > > > > > > >*)+0x24b) [0x6c76db]
  219. 3: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x317) [0x6ca987]
  220. 4: (boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<PG::RecoveryState::Primary> const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x5c) [0x6f5e8c]
  221. 5: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Active, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x90) [0x6f6900]
  222. 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x182) [0x6f6b92]
  223. 7: (boost::statechart::simple_state<PG::RecoveryState::WaitFlushedPeering, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x88) [0x6f0d98]
  224. 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0xfb) [0x6db65b]
  225. 9: (PG::RecoveryState::handle_event(boost::statechart::event_base const&, PG::RecoveryCtx*)+0x57) [0x6db8d7]
  226. 10: (PG::handle_activate_map(PG::RecoveryCtx*)+0xee) [0x69356e]
  227. 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x49d) [0x62a9cd]
  228. 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x244) [0x62ada4]
  229. 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x12) [0x666122]
  230. 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x833826]
  231. 15: (ThreadPool::WorkThread::entry()+0x10) [0x835650]
  232. 16: (()+0x7e9a) [0x7fa84ae77e9a]
  233. 17: (clone()+0x6d) [0x7fa84916873d]
  234. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  235.  
  236. --- begin dump of recent events ---
  237. -400> 2014-08-15 14:43:49.504264 7fa82a10f700 -1 ./include/interval_set.h: In function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' thread 7fa82a10f700 time 2014-08-15 14:43:49.499643
  238. ./include/interval_set.h: 391: FAILED assert(p->second >= before+len)
  239.  
  240. ceph version 0.61.9-12-ga5d0178 (a5d0178088973fdc70df692b0d874
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!