API tools faq
paste
Advertisement
Guest User

ceph OSD logs

a guest
Aug 19th, 2013
117
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 58.49 KB | None | 0 0
raw download clone embed print report
  1. root! taman:/var/log/ceph# grep -v ' pipe' osd.65.log
  2. 2013-08-19 11:07:48.478558 7f6fe367a780 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 19327
  3. 2013-08-19 11:07:48.516363 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount FIEMAP ioctl is supported and appears to work
  4. 2013-08-19 11:07:48.516380 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
  5. 2013-08-19 11:07:48.516514 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount did NOT detect btrfs
  6. 2013-08-19 11:07:48.517087 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount syscall(SYS_syncfs, fd) fully supported
  7. 2013-08-19 11:07:48.517389 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount found snaps <>
  8. 2013-08-19 11:07:49.199483 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount: enabling WRITEAHEAD journal mode: btrfs not detected
  9. 2013-08-19 11:07:52.191336 7f6fe367a780 1 journal _open /dev/sdk4 fd 18: 53687091200 bytes, block size 4096 bytes, directio = 1, aio = 1
  10. 2013-08-19 11:07:52.196020 7f6fe367a780 1 journal _open /dev/sdk4 fd 18: 53687091200 bytes, block size 4096 bytes, directio = 1, aio = 1
  11. 2013-08-19 11:07:52.196920 7f6fe367a780 1 journal close /dev/sdk4
  12. 2013-08-19 11:07:52.199908 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount FIEMAP ioctl is supported and appears to work
  13. 2013-08-19 11:07:52.199916 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
  14. 2013-08-19 11:07:52.200058 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount did NOT detect btrfs
  15. 2013-08-19 11:07:52.200886 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount syscall(SYS_syncfs, fd) fully supported
  16. 2013-08-19 11:07:52.200919 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount found snaps <>
  17. 2013-08-19 11:07:52.215850 7f6fe367a780 0 filestore(/var/lib/ceph/osd/ceph-65) mount: enabling WRITEAHEAD journal mode: btrfs not detected
  18. 2013-08-19 11:07:52.219819 7f6fe367a780 1 journal _open /dev/sdk4 fd 26: 53687091200 bytes, block size 4096 bytes, directio = 1, aio = 1
  19. 2013-08-19 11:07:52.227420 7f6fe367a780 1 journal _open /dev/sdk4 fd 26: 53687091200 bytes, block size 4096 bytes, directio = 1, aio = 1
  20. 2013-08-19 11:07:52.500342 7f6fe367a780 0 osd.65 144201 crush map has features 262144, adjusting msgr requires for clients
  21. 2013-08-19 11:07:52.500353 7f6fe367a780 0 osd.65 144201 crush map has features 262144, adjusting msgr requires for osds
  22. 2013-08-19 11:08:13.581709 7f6fbdcb5700 -1 osd/OSD.cc: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f6fbdcb5700 time 2013-08-19 11:08:13.579519
  23. osd/OSD.cc: 4844: FAILED assert(_get_map_bl(epoch, bl))
  24.  
  25. ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
  26. 1: (OSDService::get_map(unsigned int)+0x44b) [0x6f5b9b]
  27. 2: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x3c8) [0x6f8f48]
  28. 3: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x31f) [0x6f975f]
  29. 4: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x14) [0x7391d4]
  30. 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0x8f8e3a]
  31. 6: (ThreadPool::WorkThread::entry()+0x10) [0x8fa0e0]
  32. 7: (()+0x6b50) [0x7f6fe3070b50]
  33. 8: (clone()+0x6d) [0x7f6fe15cba7d]
  34. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  35.  
  36. --- begin dump of recent events ---
  37. -9752> 2013-08-19 11:08:13.336501 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.1ba( v 81'10 (0'0,81'10] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,44,70] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  38. -7827> 2013-08-19 11:08:13.396250 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.226( empty local-les=143279 n=0 ec=1 les/c 143279/143279 143278/143278/140722) [65,77,45] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  39. -5643> 2013-08-19 11:08:13.463301 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.1b9( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,44,70] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  40. -5201> 2013-08-19 11:08:13.476442 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.220( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,40,74] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  41. -4321> 2013-08-19 11:08:13.503463 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.a6( empty local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/141685) [76,65,47] r=1 lpr=144182 pi=140722-141684/1 inactive NOTIFY] state<Start>: transitioning to Stray
  42. -4320> 2013-08-19 11:08:13.503489 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.221( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,74] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  43. -4319> 2013-08-19 11:08:13.503575 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.220( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,70,42] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  44. -4315> 2013-08-19 11:08:13.503645 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.1b4( empty local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139315) [78,65,43] r=1 lpr=144182 pi=139284-141688/4 inactive NOTIFY] state<Start>: transitioning to Stray
  45. -4314> 2013-08-19 11:08:13.503646 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.90( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/128259) [41,65,71] r=1 lpr=144198 pi=127868-140721/6 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  46. -4313> 2013-08-19 11:08:13.503674 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.21f( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,70,42] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  47. -4308> 2013-08-19 11:08:13.503721 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.8f( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128259) [41,65,71] r=1 lpr=144199 pi=127868-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  48. -4305> 2013-08-19 11:08:13.503767 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.21f( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,40,74] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  49. -4304> 2013-08-19 11:08:13.503828 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.8e( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/128259) [41,65,71] r=1 lpr=144198 pi=127868-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  50. -4303> 2013-08-19 11:08:13.503882 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.21e( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,70,42] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  51. -4302> 2013-08-19 11:08:13.503925 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.85( empty local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/128259) [40,75,65] r=2 lpr=144199 pi=127368-140721/13 inactive NOTIFY] state<Start>: transitioning to Stray
  52. -4301> 2013-08-19 11:08:13.503960 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.1b5( v 81'4 (0'0,81'4] local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139315) [78,65,43] r=1 lpr=144182 pi=139284-141688/4 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  53. -4297> 2013-08-19 11:08:13.504051 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.86( empty local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/128259) [40,75,65] r=2 lpr=144199 pi=121091-140721/23 inactive NOTIFY] state<Start>: transitioning to Stray
  54. -4294> 2013-08-19 11:08:13.504166 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.1b1( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,71,41] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  55. -4292> 2013-08-19 11:08:13.504248 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.1ec( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  56. -4290> 2013-08-19 11:08:13.504274 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.87( v 81'8 (0'0,81'8] local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/128259) [40,75,65] r=2 lpr=144199 pi=127368-140721/13 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  57. -4289> 2013-08-19 11:08:13.504292 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.1b2( empty local-les=140723 n=0 ec=1 les/c 140723/140740 140722/140722/140722) [65,71,41] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  58. -4285> 2013-08-19 11:08:13.504329 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.1ee( v 81'6 (0'0,81'6] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  59. -4279> 2013-08-19 11:08:13.504414 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.1b3( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140740 140722/140722/140722) [65,71,41] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  60. -4276> 2013-08-19 11:08:13.504489 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.1ec( empty local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/139316) [75,65,48] r=1 lpr=144199 pi=139279-143274/27 inactive NOTIFY] state<Start>: transitioning to Stray
  61. -4270> 2013-08-19 11:08:13.504543 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.80( empty local-les=143279 n=0 ec=1 les/c 143279/143281 143278/143278/140722) [65,77,47] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  62. -4267> 2013-08-19 11:08:13.504585 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.1ed( v 81'10 (0'0,81'10] local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/139316) [75,65,48] r=1 lpr=144199 pi=139279-143274/27 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  63. -4266> 2013-08-19 11:08:13.504737 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.81( v 81'2 (0'0,81'2] local-les=143279 n=0 ec=1 les/c 143279/143281 143278/143278/140722) [65,77,47] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  64. -4265> 2013-08-19 11:08:13.504739 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.1b3( empty local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139315) [78,65,43] r=1 lpr=144182 pi=139284-141688/4 inactive NOTIFY] state<Start>: transitioning to Stray
  65. -4264> 2013-08-19 11:08:13.504745 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.1ed( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  66. -4258> 2013-08-19 11:08:13.504943 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.80( v 81'2 (0'0,81'2] local-les=143280 n=0 ec=1 les/c 143280/143281 143278/143278/143278) [77,46,65] r=2 lpr=144199 pi=127849-143277/22 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  67. -4257> 2013-08-19 11:08:13.504944 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.1a4( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,42,74] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  68. -4255> 2013-08-19 11:08:13.505026 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.1eb( empty local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/139316) [75,65,48] r=1 lpr=144199 pi=139279-143274/27 inactive NOTIFY] state<Start>: transitioning to Stray
  69. -4254> 2013-08-19 11:08:13.505038 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.7f( empty local-les=143280 n=0 ec=1 les/c 143280/143281 143278/143278/143278) [77,46,65] r=2 lpr=144199 pi=127950-143277/20 inactive NOTIFY] state<Start>: transitioning to Stray
  70. -4252> 2013-08-19 11:08:13.505063 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.1a6( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,42,74] r=0 lpr=144198 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  71. -4244> 2013-08-19 11:08:13.505234 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.1e0( v 133538'10 (0'0,133538'10] local-les=140723 n=1 ec=1 les/c 140723/140730 140722/140722/128260) [42,78,65] r=2 lpr=144199 pi=127371-140721/15 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  72. -4242> 2013-08-19 11:08:13.505244 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.1a5( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,42,74] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  73. -4232> 2013-08-19 11:08:13.505408 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.19c( empty local-les=142975 n=0 ec=1 les/c 142975/142975 142974/142974/142974) [45,65,71] r=1 lpr=144198 pi=142965-142973/1 inactive NOTIFY] state<Start>: transitioning to Stray
  74. -4231> 2013-08-19 11:08:13.505497 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.7f( empty local-les=143279 n=0 ec=1 les/c 143279/143279 143278/143278/140722) [65,77,47] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  75. -4230> 2013-08-19 11:08:13.505500 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.19e( v 81'4 (0'0,81'4] local-les=142975 n=0 ec=1 les/c 142975/142975 142974/142974/142974) [45,65,71] r=1 lpr=144198 pi=142965-142973/1 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  76. -4229> 2013-08-19 11:08:13.505547 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[2.1dd( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,70] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  77. -4227> 2013-08-19 11:08:13.505599 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.7e( empty local-les=143280 n=0 ec=1 les/c 143280/143281 143278/143278/143278) [77,46,65] r=2 lpr=144199 pi=127849-143277/22 inactive NOTIFY] state<Start>: transitioning to Stray
  78. -4225> 2013-08-19 11:08:13.505597 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.19d( empty local-les=142975 n=0 ec=1 les/c 142975/142975 142974/142974/142974) [45,65,71] r=1 lpr=144198 pi=142965-142973/1 inactive NOTIFY] state<Start>: transitioning to Stray
  79. -4220> 2013-08-19 11:08:13.505686 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.64( v 81'6 (0'0,81'6] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/139315) [78,65,46] r=1 lpr=144199 pi=139284-140721/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  80. -4219> 2013-08-19 11:08:13.505743 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.189( empty local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,44] r=1 lpr=144199 pi=143082-143277/1 inactive NOTIFY] state<Start>: transitioning to Stray
  81. -4218> 2013-08-19 11:08:13.505751 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.63( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139315) [78,65,46] r=1 lpr=144199 pi=139284-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  82. -4217> 2013-08-19 11:08:13.505764 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.1de( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,70] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  83. -4216> 2013-08-19 11:08:13.505828 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.18a( empty local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,44] r=1 lpr=144198 pi=143082-143277/1 inactive NOTIFY] state<Start>: transitioning to Stray
  84. -4215> 2013-08-19 11:08:13.505835 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.62( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139315) [78,65,46] r=1 lpr=144199 pi=139284-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  85. -4214> 2013-08-19 11:08:13.505936 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.18b( v 81'2 (0'0,81'2] local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,44] r=1 lpr=144199 pi=143082-143277/1 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  86. -4209> 2013-08-19 11:08:13.506008 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[0.1df( v 81'4 (0'0,81'4] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,70] r=0 lpr=144182 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  87. -4206> 2013-08-19 11:08:13.506027 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.17d( empty local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/140722) [65,76,41] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  88. -4202> 2013-08-19 11:08:13.506125 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.17e( empty local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/140722) [65,76,41] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  89. -4200> 2013-08-19 11:08:13.506134 7f6fbdcb5700 1 osd.65 pg_epoch: 144201 pg[1.1df( empty local-les=140723 n=0 ec=1 les/c 140723/140730 140722/140722/128266) [42,78,65] r=2 lpr=144198 pi=126027-140721/29 inactive NOTIFY] state<Start>: transitioning to Stray
  90. -4194> 2013-08-19 11:08:13.506169 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6804/18630 -- pg_notify(0.1e0(15),1.1df(29) epoch 144201) v4 -- ?+0 0x13107340 con 0x13025080
  91. -4193> 2013-08-19 11:08:13.506160 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.58( v 81'8 (0'0,81'8] local-les=140723 n=0 ec=1 les/c 140723/140740 140722/140722/139295) [70,65,41] r=1 lpr=144182 pi=139268-140721/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  92. -4192> 2013-08-19 11:08:13.506189 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6814/20178 -- pg_notify(2.228(5),0.22a(5),1.229(5) epoch 144201) v4 -- ?+0 0x13107180 con 0x13024f20
  93. -4191> 2013-08-19 11:08:13.506185 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.17f( v 138582'6 (0'0,138582'6] local-les=141686 n=1 ec=1 les/c 141686/141687 141685/141685/140722) [65,76,41] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  94. -4188> 2013-08-19 11:08:13.506212 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_notify(2.23e(3),1.1ec(27),0.1ed(27),2.1eb(27) epoch 144201) v4 -- ?+0 0x131068c0 con 0x20ddce0
  95. -4186> 2013-08-19 11:08:13.506268 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6808/34133 -- pg_notify(0.234(21),1.233(29),2.232(21) epoch 144201) v4 -- ?+0 0x13107dc0 con 0x13024840
  96. -4185> 2013-08-19 11:08:13.506291 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6800/48397 -- pg_query(0.1ee,0.221,1.1ed,1.220,2.1ec,2.21f epoch 144201) v2 -- ?+0 0x25f0d20 con 0x13024dc0
  97. -4184> 2013-08-19 11:08:13.506310 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.57( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,41] r=1 lpr=144198 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  98. -4183> 2013-08-19 11:08:13.506355 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6804/18630 -- pg_query(0.220,1.21f,2.21e epoch 144201) v2 -- ?+0 0x25f1c20 con 0x13025080
  99. -4182> 2013-08-19 11:08:13.506474 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6810/56558 -- pg_query(0.1df,0.228,1.1de,1.227,2.1dd,2.226 epoch 144201) v2 -- ?+0 0x135e45a0 con 0x20dde40
  100. -4181> 2013-08-19 11:08:13.506492 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6800/46656 -- pg_query(0.1df,0.220,1.1de,1.21f,2.1dd,2.21e epoch 144201) v2 -- ?+0 0x135e43c0 con 0x13024b00
  101. -4177> 2013-08-19 11:08:13.506595 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6812/64864 -- pg_query(0.221,1.220,2.21f epoch 144201) v2 -- ?+0 0x135e41e0 con 0x130242c0
  102. -4176> 2013-08-19 11:08:13.506605 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_query(0.1ee,1.1ed,2.1ec epoch 144201) v2 -- ?+0 0x135e4000 con 0x20ddce0
  103. -4174> 2013-08-19 11:08:13.506612 7f6fbdcb5700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6808/34133 -- pg_query(0.228,1.227,2.226 epoch 144201) v2 -- ?+0 0x135e4f00 con 0x13024840
  104. -4171> 2013-08-19 11:08:13.506621 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.164( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128262) [44,74,65] r=2 lpr=144182 pi=127857-140721/10 inactive NOTIFY] state<Start>: transitioning to Stray
  105. -4154> 2013-08-19 11:08:13.506805 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.56( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,41] r=1 lpr=144182 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  106. -4151> 2013-08-19 11:08:13.506858 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.166( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128262) [44,74,65] r=2 lpr=144182 pi=127857-140721/10 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  107. -4135> 2013-08-19 11:08:13.507388 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.165( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128259) [44,74,65] r=2 lpr=144198 pi=127877-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  108. -4131> 2013-08-19 11:08:13.507444 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.4c( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,44] r=1 lpr=144199 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  109. -4123> 2013-08-19 11:08:13.507542 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.4e( v 81'6 (0'0,81'6] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/139295) [70,65,44] r=1 lpr=144199 pi=139268-140721/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  110. -4117> 2013-08-19 11:08:13.507635 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.4d( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,44] r=1 lpr=144199 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  111. -4110> 2013-08-19 11:08:13.507696 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.154( v 81'4 (0'0,81'4] local-les=143276 n=0 ec=1 les/c 143276/143277 143275/143275/139307) [72,65,48] r=1 lpr=144182 pi=139272-143274/27 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  112. -4100> 2013-08-19 11:08:13.507792 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.44( v 127186'10 (0'0,127186'10] local-les=140723 n=1 ec=1 les/c 140723/140735 140722/140722/139306) [71,46,65] r=2 lpr=144199 pi=123851-140721/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  113. -4099> 2013-08-19 11:08:13.507818 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.153( empty local-les=143276 n=0 ec=1 les/c 143276/143277 143275/143275/139307) [72,65,48] r=1 lpr=144199 pi=139272-143274/27 inactive NOTIFY] state<Start>: transitioning to Stray
  114. -4095> 2013-08-19 11:08:13.507892 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.43( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/139306) [71,46,65] r=2 lpr=144199 pi=127319-140721/12 inactive NOTIFY] state<Start>: transitioning to Stray
  115. -4091> 2013-08-19 11:08:13.507942 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.152( empty local-les=143276 n=0 ec=1 les/c 143276/143277 143275/143275/139307) [72,65,48] r=1 lpr=144199 pi=139272-143274/27 inactive NOTIFY] state<Start>: transitioning to Stray
  116. -4088> 2013-08-19 11:08:13.507983 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.42( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/139306) [71,46,65] r=2 lpr=144199 pi=123851-140721/9 inactive NOTIFY] state<Start>: transitioning to Stray
  117. -4087> 2013-08-19 11:08:13.508001 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.144( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,76,45] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  118. -4085> 2013-08-19 11:08:13.508075 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.145( v 81'8 (0'0,81'8] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,76,45] r=0 lpr=144198 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  119. -4084> 2013-08-19 11:08:13.508084 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.3c( empty local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139306) [71,43,65] r=2 lpr=144199 pi=127611-141688/12 inactive NOTIFY] state<Start>: transitioning to Stray
  120. -4080> 2013-08-19 11:08:13.508222 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.140( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139306) [71,65,45] r=1 lpr=144199 pi=139270-142973/27 inactive NOTIFY] state<Start>: transitioning to Stray
  121. -4076> 2013-08-19 11:08:13.508261 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.3e( empty local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139306) [71,43,65] r=2 lpr=144199 pi=127611-141688/12 inactive NOTIFY] state<Start>: transitioning to Stray
  122. -4075> 2013-08-19 11:08:13.508290 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.142( v 81'10 (0'0,81'10] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139306) [71,65,45] r=1 lpr=144199 pi=139270-142973/27 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  123. -4068> 2013-08-19 11:08:13.508402 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.143( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,76,45] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  124. -4067> 2013-08-19 11:08:13.508402 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.3d( empty local-les=141690 n=0 ec=1 les/c 141690/141697 141689/141689/139306) [71,43,65] r=2 lpr=144199 pi=127950-141688/7 inactive NOTIFY] state<Start>: transitioning to Stray
  125. -4063> 2013-08-19 11:08:13.508534 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.141( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139306) [71,65,45] r=1 lpr=144199 pi=139270-142973/27 inactive NOTIFY] state<Start>: transitioning to Stray
  126. -4058> 2013-08-19 11:08:13.508575 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.2c( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,44,78] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  127. -4055> 2013-08-19 11:08:13.508600 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.138( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128037) [41,70,65] r=2 lpr=144198 pi=128037-140721/6 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  128. -4040> 2013-08-19 11:08:13.508742 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.137( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128037) [41,70,65] r=2 lpr=144198 pi=128037-140721/5 inactive NOTIFY] state<Start>: transitioning to Stray
  129. -4039> 2013-08-19 11:08:13.508766 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.2e( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,44,78] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  130. -4028> 2013-08-19 11:08:13.508992 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.136( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/128037) [41,70,65] r=2 lpr=144198 pi=128037-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  131. -4027> 2013-08-19 11:08:13.509012 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.2d( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,44,78] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  132. -4026> 2013-08-19 11:08:13.509051 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.131( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,41,74] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  133. -4023> 2013-08-19 11:08:13.509123 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.132( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,41,74] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  134. -4015> 2013-08-19 11:08:13.509233 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.133( v 81'6 (0'0,81'6] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/140722) [65,41,74] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  135. -4013> 2013-08-19 11:08:13.509398 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.28( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  136. -4006> 2013-08-19 11:08:13.509515 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.128( empty local-les=140729 n=0 ec=1 les/c 140729/140745 140722/140722/128259) [46,70,65] r=2 lpr=144182 pi=127857-140721/7 inactive NOTIFY] state<Start>: transitioning to Stray
  137. -4003> 2013-08-19 11:08:13.509537 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.2a( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  138. -3991> 2013-08-19 11:08:13.509697 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.29( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,75] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  139. -3985> 2013-08-19 11:08:13.509730 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.12a( v 81'2 (0'0,81'2] local-les=140729 n=0 ec=1 les/c 140729/140745 140722/140722/128259) [46,70,65] r=2 lpr=144182 pi=127857-140721/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  140. -3979> 2013-08-19 11:08:13.509830 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.28( v 81'2 (0'0,81'2] local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,45] r=1 lpr=144199 pi=143082-143277/1 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  141. -3976> 2013-08-19 11:08:13.509855 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.129( empty local-les=140729 n=0 ec=1 les/c 140729/140745 140722/140722/128259) [46,70,65] r=2 lpr=144199 pi=127611-140721/14 inactive NOTIFY] state<Start>: transitioning to Stray
  142. -3965> 2013-08-19 11:08:13.510131 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.27( empty local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,45] r=1 lpr=144199 pi=143082-143277/1 inactive NOTIFY] state<Start>: transitioning to Stray
  143. -3964> 2013-08-19 11:08:13.510239 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.26( empty local-les=143281 n=0 ec=1 les/c 143281/143281 143278/143278/143278) [77,65,45] r=1 lpr=144199 pi=143082-143277/1 inactive NOTIFY] state<Start>: transitioning to Stray
  144. -3960> 2013-08-19 11:08:13.510330 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.121( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,74,45] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  145. -3953> 2013-08-19 11:08:13.510503 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[0.20( v 81'2 (0'0,81'2] local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/140722) [65,40,76] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  146. -3937> 2013-08-19 11:08:13.510637 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[1.1f( empty local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/140722) [65,40,76] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  147. -3936> 2013-08-19 11:08:13.510664 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.122( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,74,45] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  148. -3932> 2013-08-19 11:08:13.510716 7f6fbe4b6700 1 osd.65 pg_epoch: 144201 pg[2.1e( empty local-les=141686 n=0 ec=1 les/c 141686/141687 141685/141685/140722) [65,40,76] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  149. -3928> 2013-08-19 11:08:13.510814 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6800/48397 -- pg_notify(2.85(13),1.86(23),0.87(13) epoch 144201) v4 -- ?+0 0x13107500 con 0x13024dc0
  150. -3924> 2013-08-19 11:08:13.510856 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6802/18360 -- pg_notify(0.90(6),1.8f(6),2.8e(6) epoch 144201) v4 -- ?+0 0x13106a80 con 0x20ddb80
  151. -3923> 2013-08-19 11:08:13.510894 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6800/46656 -- pg_notify(0.58(3),1.57(3),2.56(3),2.4c(3),0.4e(3),1.4d(3) epoch 144201) v4 -- ?+0 0x135da8c0 con 0x13024b00
  152. -3922> 2013-08-19 11:08:13.510909 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6802/52462 -- pg_notify(0.44(9),1.43(12),2.42(9),2.3c(12),0.3e(12),1.3d(7) epoch 144201) v4 -- ?+0 0x135da700 con 0x13024420
  153. -3921> 2013-08-19 11:08:13.510926 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.123( v 81'10 (0'0,81'10] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,74,45] r=0 lpr=144182 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  154. -3909> 2013-08-19 11:08:13.511076 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_notify(2.b4(8),0.b6(8),1.b5(56) epoch 144201) v4 -- ?+0 0x135da540 con 0x20ddce0
  155. -3907> 2013-08-19 11:08:13.511126 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6804/24167 -- pg_notify(0.a8(1),1.a7(1),2.a6(1) epoch 144201) v4 -- ?+0 0x135da380 con 0x13024000
  156. -3906> 2013-08-19 11:08:13.511153 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6808/34133 -- pg_notify(0.80(22),1.7f(20),2.7e(22),0.28(1),1.27(1),2.26(1) epoch 144201) v4 -- ?+0 0x135da1c0 con 0x13024840
  157. -3905> 2013-08-19 11:08:13.511169 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6832/5570 -- pg_notify(0.64(3),1.63(3),2.62(3) epoch 144201) v4 -- ?+0 0x135da000 con 0x130246e0
  158. -3904> 2013-08-19 11:08:13.511166 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.120( empty local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/139316) [75,65,47] r=1 lpr=144199 pi=139279-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  159. -3903> 2013-08-19 11:08:13.511185 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6800/48397 -- pg_query(0.20,0.2a,1.1f,1.29,2.1e,2.28 epoch 144201) v2 -- ?+0 0x135e50e0 con 0x13024dc0
  160. -3902> 2013-08-19 11:08:13.511203 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6808/19386 -- pg_query(0.2e,1.2d,2.2c epoch 144201) v2 -- ?+0 0x25f0000 con 0x20dd8c0
  161. -3901> 2013-08-19 11:08:13.511216 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6814/20178 -- pg_query(0.81,1.80,2.7f epoch 144201) v2 -- ?+0 0x25f1e00 con 0x13024f20
  162. -3899> 2013-08-19 11:08:13.511240 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.122( empty local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/139316) [75,65,47] r=1 lpr=144199 pi=139279-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  163. -3893> 2013-08-19 11:08:13.511324 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_query(0.2a,1.29,2.28 epoch 144201) v2 -- ?+0 0x25f1860 con 0x20ddce0
  164. -3892> 2013-08-19 11:08:13.511339 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6804/24167 -- pg_query(0.20,1.1f,2.1e epoch 144201) v2 -- ?+0 0x135e52c0 con 0x13024000
  165. -3888> 2013-08-19 11:08:13.511365 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.121( empty local-les=140724 n=0 ec=1 les/c 140724/140745 140722/140722/139316) [75,65,47] r=1 lpr=144199 pi=139279-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  166. -3887> 2013-08-19 11:08:13.511419 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6808/34133 -- pg_query(0.81,1.80,2.7f epoch 144201) v2 -- ?+0 0x25f1a40 con 0x13024840
  167. -3886> 2013-08-19 11:08:13.511434 7f6fbe4b6700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6832/5570 -- pg_query(0.2e,1.2d,2.2c epoch 144201) v2 -- ?+0 0x25f01e0 con 0x130246e0
  168. -3880> 2013-08-19 11:08:13.511484 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.120( v 81'4 (0'0,81'4] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,71,41] r=0 lpr=144198 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  169. -3858> 2013-08-19 11:08:13.511753 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.11d( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139315) [78,65,45] r=1 lpr=144182 pi=139284-142973/27 inactive NOTIFY] state<Start>: transitioning to Stray
  170. -3847> 2013-08-19 11:08:13.512022 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.11e( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139315) [78,65,45] r=1 lpr=144182 pi=139284-142973/27 inactive NOTIFY] state<Start>: transitioning to Stray
  171. -3839> 2013-08-19 11:08:13.512216 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.11f( v 81'10 (0'0,81'10] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/139315) [78,65,45] r=1 lpr=144182 pi=139284-142973/27 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  172. -3826> 2013-08-19 11:08:13.512440 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.11f( empty local-les=140723 n=0 ec=1 les/c 140723/140740 140722/140722/140722) [65,71,41] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  173. -3822> 2013-08-19 11:08:13.512513 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.11e( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,71,41] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  174. -3814> 2013-08-19 11:08:13.512604 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.115( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,46] r=1 lpr=144199 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  175. -3806> 2013-08-19 11:08:13.512658 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.116( empty local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,46] r=1 lpr=144199 pi=139268-140721/3 inactive NOTIFY] state<Start>: transitioning to Stray
  176. -3805> 2013-08-19 11:08:13.512712 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.117( v 81'6 (0'0,81'6] local-les=140723 n=0 ec=1 les/c 140723/140729 140722/140722/139295) [70,65,46] r=1 lpr=144199 pi=139268-140721/3 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  177. -3801> 2013-08-19 11:08:13.512775 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.114( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,47,70] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  178. -3794> 2013-08-19 11:08:13.512854 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.116( v 81'8 (0'0,81'8] local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,47,70] r=0 lpr=144198 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  179. -3792> 2013-08-19 11:08:13.512953 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.115( empty local-les=140723 n=0 ec=1 les/c 140723/140723 140722/140722/140722) [65,47,70] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  180. -3775> 2013-08-19 11:08:13.513300 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.108( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/133563) [42,65,72] r=1 lpr=144199 pi=127853-140721/12 inactive NOTIFY] state<Start>: transitioning to Stray
  181. -3760> 2013-08-19 11:08:13.513532 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.10a( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/133563) [42,65,72] r=1 lpr=144182 pi=127853-140721/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  182. -3757> 2013-08-19 11:08:13.513596 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.109( empty local-les=140723 n=0 ec=1 les/c 140723/140735 140722/140722/128259) [42,65,72] r=1 lpr=144199 pi=127941-140721/5 inactive NOTIFY] state<Start>: transitioning to Stray
  183. -3748> 2013-08-19 11:08:13.513726 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.fc( v 81'6 (0'0,81'6] local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/143275) [48,65,72] r=1 lpr=144198 pi=143272-143274/1 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  184. -3744> 2013-08-19 11:08:13.513792 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.fb( empty local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/143275) [48,65,72] r=1 lpr=144198 pi=143272-143274/1 inactive NOTIFY] state<Start>: transitioning to Stray
  185. -3740> 2013-08-19 11:08:13.513866 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.fa( empty local-les=143276 n=0 ec=1 les/c 143276/143276 143275/143275/143275) [48,65,72] r=1 lpr=144198 pi=143272-143274/1 inactive NOTIFY] state<Start>: transitioning to Stray
  186. -3738> 2013-08-19 11:08:13.513952 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.f0( v 81'10 (0'0,81'10] local-les=141696 n=0 ec=1 les/c 141696/141697 141689/141689/141689) [43,65,75] r=1 lpr=144198 pi=140722-141688/1 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  187. -3734> 2013-08-19 11:08:13.514012 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.ec( empty local-les=140725 n=0 ec=1 les/c 140725/140745 140722/140722/139297) [73,44,65] r=2 lpr=144198 pi=127904-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  188. -3716> 2013-08-19 11:08:13.514258 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.ef( empty local-les=141696 n=0 ec=1 les/c 141696/141697 141689/141689/141689) [43,65,75] r=1 lpr=144182 pi=140722-141688/1 inactive NOTIFY] state<Start>: transitioning to Stray
  189. -3706> 2013-08-19 11:08:13.514347 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.ee( v 81'2 (0'0,81'2] local-les=140725 n=0 ec=1 les/c 140725/140745 140722/140722/139297) [73,44,65] r=2 lpr=144199 pi=127904-140721/6 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  190. -3698> 2013-08-19 11:08:13.514544 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.ee( empty local-les=141696 n=0 ec=1 les/c 141696/141697 141689/141689/141689) [43,65,75] r=1 lpr=144182 pi=140722-141688/1 inactive NOTIFY] state<Start>: transitioning to Stray
  191. -3694> 2013-08-19 11:08:13.514607 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.ed( empty local-les=140725 n=0 ec=1 les/c 140725/140745 140722/140722/139297) [73,44,65] r=2 lpr=144198 pi=127904-140721/6 inactive NOTIFY] state<Start>: transitioning to Stray
  192. -3684> 2013-08-19 11:08:13.514799 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.e4( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,73] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  193. -3683> 2013-08-19 11:08:13.514853 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.e6( v 81'2 (0'0,81'2] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,73] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  194. -3679> 2013-08-19 11:08:13.514937 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.e5( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/140722) [65,45,73] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  195. -3670> 2013-08-19 11:08:13.515001 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.e0( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,44] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  196. -3666> 2013-08-19 11:08:13.515055 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.e2( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,44] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  197. -3659> 2013-08-19 11:08:13.515108 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.e1( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,44] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  198. -3643> 2013-08-19 11:08:13.515477 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.dc( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,74] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  199. -3632> 2013-08-19 11:08:13.515551 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.de( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,74] r=0 lpr=144198 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  200. -3628> 2013-08-19 11:08:13.515740 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.dd( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,40,74] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  201. -3603> 2013-08-19 11:08:13.516128 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.d0( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,47] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  202. -3599> 2013-08-19 11:08:13.516289 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.d2( v 81'2 (0'0,81'2] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,47] r=0 lpr=144182 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  203. -3588> 2013-08-19 11:08:13.516396 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.d1( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,75,47] r=0 lpr=144199 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  204. -3578> 2013-08-19 11:08:13.516676 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.c4( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,44,78] r=0 lpr=144182 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  205. -3565> 2013-08-19 11:08:13.516753 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.c6( v 81'8 (0'0,81'8] local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,44,78] r=0 lpr=144199 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  206. -3559> 2013-08-19 11:08:13.516911 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.c5( empty local-les=140723 n=0 ec=1 les/c 140723/140745 140722/140722/140722) [65,44,78] r=0 lpr=144198 mlcod 0'0 inactive] state<Start>: transitioning to Primary
  207. -3552> 2013-08-19 11:08:13.517075 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[2.bd( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/142974) [45,78,65] r=2 lpr=144199 pi=127857-142973/32 inactive NOTIFY] state<Start>: transitioning to Stray
  208. -3548> 2013-08-19 11:08:13.517181 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[1.be( empty local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/142974) [45,78,65] r=2 lpr=144199 pi=127877-142973/31 inactive NOTIFY] state<Start>: transitioning to Stray
  209. -3535> 2013-08-19 11:08:13.517267 7f6fbd4b4700 1 osd.65 pg_epoch: 144201 pg[0.bf( v 81'6 (0'0,81'6] local-les=142975 n=0 ec=1 les/c 142975/142976 142974/142974/142974) [45,78,65] r=2 lpr=144199 pi=127857-142973/32 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
  210. -3531> 2013-08-19 11:08:13.517350 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6802/18360 -- pg_notify(0.138(6),1.137(5),2.136(6) epoch 144201) v4 -- ?+0 0x135dae00 con 0x20ddb80
  211. -3530> 2013-08-19 11:08:13.517364 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6804/18630 -- pg_notify(2.108(12),0.10a(12),1.109(5) epoch 144201) v4 -- ?+0 0x135dac40 con 0x13025080
  212. -3529> 2013-08-19 11:08:13.517372 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6816/33735 -- pg_notify(0.f0(1),1.ef(1),2.ee(1) epoch 144201) v4 -- ?+0 0x135daa80 con 0x20dd760
  213. -3528> 2013-08-19 11:08:13.517382 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6808/19386 -- pg_notify(2.164(10),0.166(10),1.165(6) epoch 144201) v4 -- ?+0 0x13106540 con 0x20dd8c0
  214. -3526> 2013-08-19 11:08:13.517391 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6810/56558 -- pg_notify(2.19c(1),0.19e(1),1.19d(1),2.bd(32),1.be(31),0.bf(32) epoch 144201) v4 -- ?+0 0x13106fc0 con 0x20dde40
  215. -3524> 2013-08-19 11:08:13.517403 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6806/19916 -- pg_notify(2.128(7),0.12a(7),1.129(14) epoch 144201) v4 -- ?+0 0x13106c40 con 0x13024160
  216. -3522> 2013-08-19 11:08:13.517416 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6818/48371 -- pg_notify(0.fc(1),1.fb(1),2.fa(1) epoch 144201) v4 -- ?+0 0x2249a40 con 0x13024c60
  217. -3521> 2013-08-19 11:08:13.517422 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6800/46656 -- pg_notify(2.115(3),1.116(3),0.117(3) epoch 144201) v4 -- ?+0 0x2249500 con 0x13024b00
  218. -3520> 2013-08-19 11:08:13.517429 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6802/52462 -- pg_notify(2.140(27),0.142(27),1.141(27) epoch 144201) v4 -- ?+0 0x2249c00 con 0x13024420
  219. -3519> 2013-08-19 11:08:13.517436 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6806/57779 -- pg_notify(0.154(27),1.153(27),2.152(27) epoch 144201) v4 -- ?+0 0x22496c0 con 0x130249a0
  220. -3518> 2013-08-19 11:08:13.517444 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6810/60430 -- pg_notify(2.ec(6),0.ee(6),1.ed(6) epoch 144201) v4 -- ?+0 0x2248fc0 con 0x13024580
  221. -3511> 2013-08-19 11:08:13.517536 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6812/64864 -- pg_notify(2.1b9(27),1.1ba(27),0.1bb(27) epoch 144201) v4 -- ?+0 0x2248380 con 0x130242c0
  222. -3510> 2013-08-19 11:08:13.517551 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_notify(2.120(3),0.122(3),1.121(3) epoch 144201) v4 -- ?+0 0x2249dc0 con 0x20ddce0
  223. -3509> 2013-08-19 11:08:13.517560 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6808/34133 -- pg_notify(2.189(1),1.18a(1),0.18b(1) epoch 144201) v4 -- ?+0 0x2248540 con 0x13024840
  224. -3508> 2013-08-19 11:08:13.517568 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6832/5570 -- pg_notify(1.1b4(4),0.1b5(4),2.1b3(4),2.11d(27),1.11e(27),0.11f(27) epoch 144201) v4 -- ?+0 0x13107dc0 con 0x130246e0
  225. -3507> 2013-08-19 11:08:13.517577 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6800/48397 -- pg_query(0.de,1.dd,2.dc epoch 144201) v2 -- ?+0 0x135e5c20 con 0x13024dc0
  226. -3506> 2013-08-19 11:08:13.517584 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6802/18360 -- pg_query(0.120,0.133,0.17f,0.1b3,1.11f,1.132,1.17e,1.1b2,2.11e,2.131,2.17d,2.1b1 epoch 144201) v2 -- ?+0 0x135e4f00 con 0x20ddb80
  227. -3504> 2013-08-19 11:08:13.517601 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6804/18630 -- pg_query(0.1a6,1.1a5,2.1a4 epoch 144201) v2 -- ?+0 0x13648960 con 0x13025080
  228. -3502> 2013-08-19 11:08:13.517611 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6816/33735 -- pg_query(0.1bf,1.1be,2.1bd epoch 144201) v2 -- ?+0 0x13648780 con 0x20dd760
  229. -3501> 2013-08-19 11:08:13.517626 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6808/19386 -- pg_query(0.c6,0.e2,0.1ba,1.c5,1.e1,1.1b9,2.c4,2.e0,2.1b8 epoch 144201) v2 -- ?+0 0x136485a0 con 0x20dd8c0
  230. -3495> 2013-08-19 11:08:13.517655 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6810/56558 -- pg_query(0.e6,0.123,0.145,1.e5,1.122,1.144,2.e4,2.121,2.143 epoch 144201) v2 -- ?+0 0x136483c0 con 0x20dde40
  231. -3492> 2013-08-19 11:08:13.517704 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.9:6814/20178 -- pg_query(0.d2,0.116,1.d1,1.115,2.d0,2.114 epoch 144201) v2 -- ?+0 0x136481e0 con 0x13024f20
  232. -3488> 2013-08-19 11:08:13.517748 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6800/46656 -- pg_query(0.116,0.1ba,0.1bf,1.115,1.1b9,1.1be,2.114,2.1b8,2.1bd epoch 144201) v2 -- ?+0 0x13648000 con 0x13024b00
  233. -3480> 2013-08-19 11:08:13.517885 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6802/52462 -- pg_query(0.120,0.1b3,1.11f,1.1b2,2.11e,2.1b1 epoch 144201) v2 -- ?+0 0x136494a0 con 0x13024420
  234. -3478> 2013-08-19 11:08:13.517911 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6810/60430 -- pg_query(0.e6,1.e5,2.e4 epoch 144201) v2 -- ?+0 0x136492c0 con 0x13024580
  235. -3476> 2013-08-19 11:08:13.517925 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6812/64864 -- pg_query(0.de,0.123,0.133,0.1a6,1.dd,1.122,1.132,1.1a5,2.dc,2.121,2.131,2.1a4 epoch 144201) v2 -- ?+0 0x136490e0 con 0x130242c0
  236. -3474> 2013-08-19 11:08:13.517946 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6816/2868 -- pg_query(0.d2,0.e2,1.d1,1.e1,2.d0,2.e0 epoch 144201) v2 -- ?+0 0x13648f00 con 0x20ddce0
  237. -3473> 2013-08-19 11:08:13.517969 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6804/24167 -- pg_query(0.145,0.17f,1.144,1.17e,2.143,2.17d epoch 144201) v2 -- ?+0 0x13648d20 con 0x13024000
  238. -3471> 2013-08-19 11:08:13.517979 7f6fbd4b4700 1 -- 192.168.42.10:6803/19327 --> 192.168.42.12:6832/5570 -- pg_query(0.c6,1.c5,2.c4 epoch 144201) v2 -- ?+0 0x13648b40 con 0x130246e0
  239. -3408> 2013-08-19 11:08:13.519005 7f6fe367a780 10 monclient: get_version osdmap
  240. -3407> 2013-08-19 11:08:13.519010 7f6fe367a780 10 monclient: _send_mon_message to mon.rurkh at 10.0.0.19:6789/0
  241. -3406> 2013-08-19 11:08:13.519015 7f6fe367a780 1 -- 10.0.0.10:6801/19327 --> 10.0.0.19:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x211cea0 con 0x20dd080
  242. -3369> 2013-08-19 11:08:13.519697 7f6fc44c2700 1 -- 10.0.0.10:6801/19327 <== mon.2 10.0.0.19:6789/0 10 ==== mon_check_map_ack(handle=1 version=144978) v2 ==== 24+0+0 (3438144437 0 0) 0x2248c40 con 0x20dd080
  243. -3368> 2013-08-19 11:08:13.519747 7f6fc04ba700 10 monclient: renew_subs
  244. -3367> 2013-08-19 11:08:13.519756 7f6fc04ba700 10 monclient: _send_mon_message to mon.rurkh at 10.0.0.19:6789/0
  245. -3366> 2013-08-19 11:08:13.519769 7f6fc04ba700 1 -- 10.0.0.10:6801/19327 --> 10.0.0.19:6789/0 -- mon_subscribe({monmap=16+,osd_pg_creates=0,osdmap=144278}) v2 -- ?+0 0x13107340 con 0x20dd080
  246. -3273> 2013-08-19 11:08:13.521441 7f6fc44c2700 1 -- 10.0.0.10:6801/19327 <== mon.2 10.0.0.19:6789/0 11 ==== osd_map(144279..144279 src has 144279..144978) v3 ==== 46457+0+0 (4151115210 0 0) 0x130b8d80 con 0x20dd080
  247. -3272> 2013-08-19 11:08:13.521467 7f6fc44c2700 3 osd.65 144201 handle_osd_map epochs [144279,144279], i have 144201, src has [144279,144978]
  248. 0> 2013-08-19 11:08:13.581709 7f6fbdcb5700 -1 osd/OSD.cc: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f6fbdcb5700 time 2013-08-19 11:08:13.579519
  249. osd/OSD.cc: 4844: FAILED assert(_get_map_bl(epoch, bl))
  250.  
  251. ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
  252. 1: (OSDService::get_map(unsigned int)+0x44b) [0x6f5b9b]
  253. 2: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x3c8) [0x6f8f48]
  254. 3: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x31f) [0x6f975f]
  255. 4: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x14) [0x7391d4]
  256. 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0x8f8e3a]
  257. 6: (ThreadPool::WorkThread::entry()+0x10) [0x8fa0e0]
  258. 7: (()+0x6b50) [0x7f6fe3070b50]
  259. 8: (clone()+0x6d) [0x7f6fe15cba7d]
  260. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  261.  
  262. --- logging levels ---
  263. 0/ 5 none
  264. 0/ 1 lockdep
  265. 0/ 1 context
  266. 1/ 1 crush
  267. 1/ 5 mds
  268. 1/ 5 mds_balancer
  269. 1/ 5 mds_locker
  270. 1/ 5 mds_log
  271. 1/ 5 mds_log_expire
  272. 1/ 5 mds_migrator
  273. 0/ 1 buffer
  274. 0/ 1 timer
  275. 0/ 1 filer
  276. 0/ 1 striper
  277. 0/ 1 objecter
  278. 0/ 5 rados
  279. 0/ 5 rbd
  280. 0/ 5 journaler
  281. 0/ 5 objectcacher
  282. 0/ 5 client
  283. 0/ 5 osd
  284. 0/ 5 optracker
  285. 0/ 5 objclass
  286. 1/ 3 filestore
  287. 1/ 3 journal
  288. 0/ 5 ms
  289. 1/ 5 mon
  290. 0/10 monc
  291. 0/ 5 paxos
  292. 0/ 5 tp
  293. 1/ 5 auth
  294. 1/ 5 crypto
  295. 1/ 1 finisher
  296. 1/ 5 heartbeatmap
  297. 1/ 5 perfcounter
  298. 1/ 5 rgw
  299. 1/ 5 hadoop
  300. 1/ 5 javaclient
  301. 1/ 5 asok
  302. 1/ 1 throttle
  303. -1/-1 (syslog threshold)
  304. -1/-1 (stderr threshold)
  305. max_recent 10000
  306. max_new 1000
  307. log_file /var/log/ceph/osd.65.log
  308. --- end dump of recent events ---
  309. 2013-08-19 11:08:13.665511 7f6fbdcb5700 -1 *** Caught signal (Aborted) **
  310. in thread 7f6fbdcb5700
  311.  
  312. ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
  313. 1: /usr/bin/ceph-osd() [0x855d62]
  314. 2: (()+0xf030) [0x7f6fe3079030]
  315. 3: (gsignal()+0x35) [0x7f6fe1523475]
  316. 4: (abort()+0x180) [0x7f6fe15266f0]
  317. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f6fe1d7889d]
  318. 6: (()+0x63996) [0x7f6fe1d76996]
  319. 7: (()+0x639c3) [0x7f6fe1d769c3]
  320. 8: (()+0x63bee) [0x7f6fe1d76bee]
  321. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x9022ea]
  322. 10: (OSDService::get_map(unsigned int)+0x44b) [0x6f5b9b]
  323. 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x3c8) [0x6f8f48]
  324. 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x31f) [0x6f975f]
  325. 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x14) [0x7391d4]
  326. 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0x8f8e3a]
  327. 15: (ThreadPool::WorkThread::entry()+0x10) [0x8fa0e0]
  328. 16: (()+0x6b50) [0x7f6fe3070b50]
  329. 17: (clone()+0x6d) [0x7f6fe15cba7d]
  330. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  331.  
  332. --- begin dump of recent events ---
  333. 0> 2013-08-19 11:08:13.665511 7f6fbdcb5700 -1 *** Caught signal (Aborted) **
  334. in thread 7f6fbdcb5700
  335.  
  336. ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
  337. 1: /usr/bin/ceph-osd() [0x855d62]
  338. 2: (()+0xf030) [0x7f6fe3079030]
  339. 3: (gsignal()+0x35) [0x7f6fe1523475]
  340. 4: (abort()+0x180) [0x7f6fe15266f0]
  341. 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f6fe1d7889d]
  342. 6: (()+0x63996) [0x7f6fe1d76996]
  343. 7: (()+0x639c3) [0x7f6fe1d769c3]
  344. 8: (()+0x63bee) [0x7f6fe1d76bee]
  345. 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x9022ea]
  346. 10: (OSDService::get_map(unsigned int)+0x44b) [0x6f5b9b]
  347. 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x3c8) [0x6f8f48]
  348. 12: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x31f) [0x6f975f]
  349. 13: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x14) [0x7391d4]
  350. 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0x8f8e3a]
  351. 15: (ThreadPool::WorkThread::entry()+0x10) [0x8fa0e0]
  352. 16: (()+0x6b50) [0x7f6fe3070b50]
  353. 17: (clone()+0x6d) [0x7f6fe15cba7d]
  354. NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  355.  
  356. --- logging levels ---
  357. 0/ 5 none
  358. 0/ 1 lockdep
  359. 0/ 1 context
  360. 1/ 1 crush
  361. 1/ 5 mds
  362. 1/ 5 mds_balancer
  363. 1/ 5 mds_locker
  364. 1/ 5 mds_log
  365. 1/ 5 mds_log_expire
  366. 1/ 5 mds_migrator
  367. 0/ 1 buffer
  368. 0/ 1 timer
  369. 0/ 1 filer
  370. 0/ 1 striper
  371. 0/ 1 objecter
  372. 0/ 5 rados
  373. 0/ 5 rbd
  374. 0/ 5 journaler
  375. 0/ 5 objectcacher
  376. 0/ 5 client
  377. 0/ 5 osd
  378. 0/ 5 optracker
  379. 0/ 5 objclass
  380. 1/ 3 filestore
  381. 1/ 3 journal
  382. 0/ 5 ms
  383. 1/ 5 mon
  384. 0/10 monc
  385. 0/ 5 paxos
  386. 0/ 5 tp
  387. 1/ 5 auth
  388. 1/ 5 crypto
  389. 1/ 1 finisher
  390. 1/ 5 heartbeatmap
  391. 1/ 5 perfcounter
  392. 1/ 5 rgw
  393. 1/ 5 hadoop
  394. 1/ 5 javaclient
  395. 1/ 5 asok
  396. 1/ 1 throttle
  397. -1/-1 (syslog threshold)
  398. -1/-1 (stderr threshold)
  399. max_recent 10000
  400. max_new 1000
  401. log_file /var/log/ceph/osd.65.log
  402. --- end dump of recent events ---
  403. root! taman:/var/log/ceph#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!