API tools faq
paste
Advertisement
Guest User

ceph-osd.38.log

a guest
Apr 1st, 2013
181
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 45.74 KB | None | 0 0
raw download clone embed print report
  1. 2013-03-31 16:42:35.274469 7fbe6f51d700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  2. 2013-03-31 16:42:35.567857 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  3. 2013-03-31 16:42:35.691060 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  4. 2013-03-31 16:42:35.711695 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  5. 2013-03-31 16:42:35.889407 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  6. 2013-03-31 16:42:35.942686 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  7. 2013-03-31 16:42:35.984576 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  8. 2013-03-31 16:42:36.305241 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  9. 2013-03-31 16:42:36.668296 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  10. 2013-03-31 16:42:36.970596 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  11. 2013-03-31 16:42:37.056681 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  12. 2013-03-31 16:42:37.085481 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  13. 2013-03-31 16:42:37.245347 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  14. 2013-03-31 16:42:37.391804 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  15. 2013-03-31 16:42:37.436255 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  16. 2013-03-31 16:42:37.557444 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  17. 2013-03-31 16:42:37.714040 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  18. 2013-03-31 16:42:37.829259 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  19. 2013-03-31 16:42:37.914404 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  20. 2013-03-31 16:42:37.937709 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  21. 2013-03-31 16:42:38.006666 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  22. 2013-03-31 16:42:38.071506 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  23. 2013-03-31 16:42:38.190357 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  24. 2013-03-31 16:42:38.415241 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  25. 2013-03-31 16:42:38.930746 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  26. 2013-03-31 16:42:39.038252 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  27. 2013-03-31 16:42:39.256430 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  28. 2013-03-31 16:42:39.265846 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  29. 2013-03-31 16:42:39.443969 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  30. 2013-03-31 16:42:39.529242 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  31. 2013-03-31 16:42:39.757058 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  32. 2013-03-31 16:42:39.766501 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  33. 2013-03-31 16:42:39.812268 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  34. 2013-03-31 16:42:39.986066 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  35. 2013-03-31 16:42:40.145860 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  36. 2013-03-31 16:42:40.274555 7fbe6f51d700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  37. 2013-03-31 16:42:40.614738 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  38. 2013-03-31 16:42:40.646264 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  39. 2013-03-31 16:42:40.768744 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  40. 2013-03-31 16:42:40.912785 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  41. 2013-03-31 16:42:41.146733 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  42. 2013-03-31 16:42:41.715254 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  43. 2013-03-31 16:42:42.469207 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  44. 2013-03-31 16:42:42.538836 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  45. 2013-03-31 16:42:42.692577 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  46. 2013-03-31 16:42:42.944499 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  47. 2013-03-31 16:42:43.115703 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  48. 2013-03-31 16:42:43.193179 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  49. 2013-03-31 16:42:43.307129 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  50. 2013-03-31 16:42:43.458340 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  51. 2013-03-31 16:42:43.631062 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  52. 2013-03-31 16:42:43.868785 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  53. 2013-03-31 16:42:43.972010 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  54. 2013-03-31 16:42:44.015855 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  55. 2013-03-31 16:42:44.090935 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  56. 2013-03-31 16:42:44.131654 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  57. 2013-03-31 16:42:44.408194 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  58. 2013-03-31 16:42:44.457870 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  59. 2013-03-31 16:42:44.632629 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  60. 2013-03-31 16:42:44.769643 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  61. 2013-03-31 16:42:45.191555 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  62. 2013-03-31 16:42:45.274664 7fbe6f51d700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  63. 2013-03-31 16:42:45.286852 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  64. 2013-03-31 16:42:45.416405 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  65. 2013-03-31 16:42:45.429914 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  66. 2013-03-31 16:42:45.508901 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  67. 2013-03-31 16:42:46.009745 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  68. 2013-03-31 16:42:46.094175 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  69. 2013-03-31 16:42:46.213106 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  70. 2013-03-31 16:42:46.447199 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  71. 2013-03-31 16:42:46.595194 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  72. 2013-03-31 16:42:46.916589 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  73. 2013-03-31 16:42:46.933167 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  74. 2013-03-31 16:42:47.359450 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  75. 2013-03-31 16:42:47.547697 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  76. 2013-03-31 16:42:47.559056 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  77. 2013-03-31 16:42:47.671920 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  78. 2013-03-31 16:42:48.187443 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  79. 2013-03-31 16:42:48.439473 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  80. 2013-03-31 16:42:48.569337 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  81. 2013-03-31 16:42:48.688173 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  82. 2013-03-31 16:42:48.845192 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  83. 2013-03-31 16:42:48.930598 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  84. 2013-03-31 16:42:48.941092 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  85. 2013-03-31 16:42:49.292500 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  86. 2013-03-31 16:42:49.511014 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  87. 2013-03-31 16:42:49.859578 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  88. 2013-03-31 16:42:49.872790 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  89. 2013-03-31 16:42:50.013598 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  90. 2013-03-31 16:42:50.031474 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  91. 2013-03-31 16:42:50.274748 7fbe6f51d700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  92. 2013-03-31 16:42:50.313623 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  93. 2013-03-31 16:42:50.360580 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  94. 2013-03-31 16:42:50.641577 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  95. 2013-03-31 16:42:50.814078 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  96. 2013-03-31 16:42:50.861629 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  97. 2013-03-31 16:42:51.295802 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  98. 2013-03-31 16:42:51.316908 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  99. 2013-03-31 16:42:51.460124 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  100. 2013-03-31 16:42:51.588968 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  101. 2013-03-31 16:42:51.633954 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  102. 2013-03-31 16:42:51.647921 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  103. 2013-03-31 16:42:51.772362 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  104. 2013-03-31 16:42:52.069883 7fbe63505700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  105. 2013-03-31 16:42:52.092525 7fbe61501700 1 heartbeat_map reset_timeout 'OSD::op_tp thread 0x7fbe61501700' had timed out after 15
  106. 2013-03-31 16:42:52.093424 7fbe6dd1a700 0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 32.422904 secs
  107. 2013-03-31 16:42:52.093435 7fbe6dd1a700 0 log [WRN] : slow request 32.422904 seconds old, received at 2013-03-31 16:42:19.670462: osd_sub_op(client.7203.0:5883239 17.63 959ef4e3/rbd_data.1e892ae8944a.0000000000009be8/head//17 [] v 5020'190061 snapset=0=[]:[] snapc=0=[]) v7 currently reached pg
  108. 2013-03-31 16:42:52.093446 7fbe6dd1a700 0 log [WRN] : slow request 32.354784 seconds old, received at 2013-03-31 16:42:19.738582: osd_op(client.7197.0:6616157 rbd_data.1c1a2ae8944a.0000000000003257 [write 1458176~8192] 17.c38ed697) v4 currently waiting for subops from [46]
  109. 2013-03-31 16:42:52.394372 7fbe5779c700 0 -- 172.22.24.14:6825/7800 >> 172.22.24.15:6836/26299 pipe(0xb58aa00 sd=179 :6825 s=2 pgs=7651 cs=37 l=0).fault with nothing to send, going to standby
  110. 2013-03-31 16:42:52.402265 7fbe592b7700 0 -- 172.22.24.14:6825/7800 >> 172.22.24.19:6809/19837 pipe(0x3d0a500 sd=45 :6825 s=2 pgs=20374 cs=1 l=0).fault with nothing to send, going to standby
  111. 2013-03-31 16:42:52.411583 7fbe586ab700 0 -- 172.22.24.14:6825/7800 >> 172.22.24.11:6817/26776 pipe(0x3fec500 sd=117 :47038 s=2 pgs=5743 cs=41 l=0).fault with nothing to send, going to standby
  112. 2013-03-31 16:42:52.411613 7fbe5c4e9700 0 -- 172.22.24.14:6825/7800 >> 172.22.24.11:6847/17505 pipe(0x326a780 sd=49 :52605 s=2 pgs=6642 cs=39 l=0).fault with nothing to send, going to standby
  113. 2013-03-31 16:42:52.414666 7fbe55e83700 0 -- 172.22.24.14:6825/7800 >> 172.22.24.20:6808/3009 pipe(0x76ab280 sd=38 :40204 s=2 pgs=7361 cs=1 l=0).fault with nothing to send, going to standby
  114. 2013-03-31 16:42:52.421600 7fbe65d0a700 0 log [WRN] : map e5030 wrongly marked me down
  115. 2013-03-31 16:42:52.487655 7fbe583a8700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :6825 s=2 pgs=9322 cs=3 l=0).fault with nothing to send, going to standby
  116. 2013-03-31 16:42:52.597411 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 1 <= 82968 0x3220e00 osd_map(5024..5030 src has 310..5030) v3, discarding
  117. 2013-03-31 16:42:52.605357 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 2 <= 82968 0x3220e00 osd_map(5030..5030 src has 310..5030) v3, discarding
  118. 2013-03-31 16:42:52.660251 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 3 <= 82968 0x3220e00 osd_map(5030..5030 src has 310..5030) v3, discarding
  119. 2013-03-31 16:42:53.182438 7fbe5375c700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.16:6801/3562 pipe(0x3931780 sd=28 :46555 s=2 pgs=33880 cs=1 l=0).fault with nothing to send, going to standby
  120. 2013-03-31 16:42:53.384312 7fbe582a7700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.11:6810/18376 pipe(0xb364280 sd=180 :6825 s=2 pgs=6380 cs=51 l=0).fault with nothing to send, going to standby
  121. 2013-03-31 16:42:53.477492 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 4 <= 82968 0x649b400 osd_map(5030..5031 src has 310..5031) v3, discarding
  122. 2013-03-31 16:42:53.477863 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 5 <= 82968 0x437a000 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3) epoch 5031) v4, discarding
  123. 2013-03-31 16:42:53.478011 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 6 <= 82968 0x437a000 pg_query(10.6a,10.2bf,10.3b9,16.64,17.63 epoch 5031) v2, discarding
  124. 2013-03-31 16:42:53.530593 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 7 <= 82968 0x9dbbe00 pg_notify(0.28(3) epoch 5031) v4, discarding
  125. 2013-03-31 16:42:53.530719 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 8 <= 82968 0x9dbbe00 pg_notify(1.27(3) epoch 5031) v4, discarding
  126. 2013-03-31 16:42:53.530823 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 9 <= 82968 0x9dbbe00 pg_notify(16.18(3) epoch 5031) v4, discarding
  127. 2013-03-31 16:42:53.530943 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 10 <= 82968 0x9dbbe00 pg_notify(17.17(3) epoch 5031) v4, discarding
  128. 2013-03-31 16:42:53.531119 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 11 <= 82968 0x9dbbe00 pg_notify(2.26(3),10.1e(3),11.1d(3),12.1c(3),13.1b(3),14.1a(3),15.19(3) epoch 5031) v4, discarding
  129. 2013-03-31 16:42:53.921055 7fbe582a7700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.11:6810/18376 pipe(0xb364280 sd=100 :57294 s=2 pgs=6381 cs=53 l=0).reader got old message 1 <= 450 0x8945600 osd_map(5030..5031 src has 310..5031) v3, discarding
  130. 2013-03-31 16:42:53.921167 7fbe582a7700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.11:6810/18376 pipe(0xb364280 sd=100 :57294 s=2 pgs=6381 cs=53 l=0).reader got old message 2 <= 450 0x9dbb540 pg_query(10.214 epoch 5031) v2, discarding
  131. 2013-03-31 16:42:54.639349 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 12 <= 82968 0x649b400 osd_map(5032..5032 src has 310..5032) v3, discarding
  132. 2013-03-31 16:42:54.639519 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 13 <= 82968 0x558a380 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5032) v4, discarding
  133. 2013-03-31 16:42:54.639632 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 14 <= 82968 0x558a380 pg_info(2 pgs e5032:16.64(2),10.2bf(2)) v3, discarding
  134. 2013-03-31 16:42:54.642351 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 15 <= 82968 0x558a380 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5032) v4, discarding
  135. 2013-03-31 16:42:54.642476 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 16 <= 82968 0x558a380 pg_info(1 pgs e5032:10.6a(2)) v3, discarding
  136. 2013-03-31 16:42:54.947538 7fbe582a7700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.11:6810/18376 pipe(0xb364280 sd=100 :57294 s=2 pgs=6381 cs=53 l=0).reader got old message 3 <= 450 0x8945600 osd_map(5032..5032 src has 310..5032) v3, discarding
  137. 2013-03-31 16:42:54.947654 7fbe582a7700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.11:6810/18376 pipe(0xb364280 sd=100 :57294 s=2 pgs=6381 cs=53 l=0).reader got old message 4 <= 450 0x9dbb540 pg_info(1 pgs e5032:10.214(2)) v3, discarding
  138. 2013-03-31 16:42:55.002691 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 17 <= 82968 0xbd19e00 pg_notify(10.3b9(5) epoch 5032) v4, discarding
  139. 2013-03-31 16:42:55.856846 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 18 <= 82968 0xcd88e00 osd_map(5033..5033 src has 310..5033) v3, discarding
  140. 2013-03-31 16:42:55.857261 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 19 <= 82968 0xbd19e00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5033) v4, discarding
  141. 2013-03-31 16:42:56.608027 7fbe60d00700 0 log [INF] : 10.3be restarting backfill on osd.56 from (0'0,0'0] MAX to 5018'61680
  142. 2013-03-31 16:42:57.193314 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 20 <= 82968 0xcd88e00 osd_map(5034..5034 src has 310..5034) v3, discarding
  143. 2013-03-31 16:42:57.193415 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 21 <= 82968 0x3f12e00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3) epoch 5034) v4, discarding
  144. 2013-03-31 16:42:57.195135 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 22 <= 82968 0x3f12e00 pg_notify(15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5034) v4, discarding
  145. 2013-03-31 16:42:57.988486 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 23 <= 82968 0x31f8200 osd_map(5035..5035 src has 310..5035) v3, discarding
  146. 2013-03-31 16:42:57.988582 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 24 <= 82968 0x486f000 pg_notify(0.28(3),1.27(3) epoch 5035) v4, discarding
  147. 2013-03-31 16:42:58.000037 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 25 <= 82968 0x486f000 pg_notify(2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5035) v4, discarding
  148. 2013-03-31 16:42:58.324118 7fbe55479700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.19:6814/19507 pipe(0xa666c80 sd=445 :6830 s=2 pgs=87576 cs=1 l=0).fault with nothing to send, going to standby
  149. 2013-03-31 16:42:58.923721 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 26 <= 82968 0x31f8200 osd_map(5036..5036 src has 310..5036) v3, discarding
  150. 2013-03-31 16:42:58.924144 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 27 <= 82968 0x9b19e00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3) epoch 5036) v4, discarding
  151. 2013-03-31 16:42:58.924847 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 28 <= 82968 0x9b19e00 pg_notify(12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5036) v4, discarding
  152. 2013-03-31 16:43:02.520727 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 29 <= 82968 0x50cee00 osd_map(5037..5037 src has 310..5037) v3, discarding
  153. 2013-03-31 16:43:02.520867 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 30 <= 82968 0x887d540 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5037) v4, discarding
  154. 2013-03-31 16:43:02.549980 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 31 <= 82968 0x887d540 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5037) v4, discarding
  155. 2013-03-31 16:43:03.597923 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 32 <= 82968 0x701ee00 osd_map(5038..5038 src has 310..5038) v3, discarding
  156. 2013-03-31 16:43:03.598371 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 33 <= 82968 0x887de00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5038) v4, discarding
  157. 2013-03-31 16:43:08.643925 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 34 <= 82968 0x6b92c00 osd_map(5039..5039 src has 310..5039) v3, discarding
  158. 2013-03-31 16:43:08.644106 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 35 <= 82968 0x955b380 pg_notify(2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3) epoch 5039) v4, discarding
  159. 2013-03-31 16:43:08.655041 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 36 <= 82968 0x955b380 pg_notify(0.28(3),1.27(3) epoch 5039) v4, discarding
  160. 2013-03-31 16:43:08.657392 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 37 <= 82968 0x955b380 pg_notify(12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5039) v4, discarding
  161. 2013-03-31 16:43:09.548300 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 38 <= 82968 0x6b92c00 osd_map(5040..5040 src has 310..5040) v3, discarding
  162. 2013-03-31 16:43:09.548444 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 39 <= 82968 0x955b380 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3) epoch 5040) v4, discarding
  163. 2013-03-31 16:43:09.571856 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 40 <= 82968 0x955b380 pg_notify(10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5040) v4, discarding
  164. 2013-03-31 16:43:09.633457 7fbe61501700 0 log [INF] : 10.3ae restarting backfill on osd.30 from (0'0,0'0] MAX to 5018'61389
  165. 2013-03-31 16:43:10.628935 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 41 <= 82968 0x4271000 osd_map(5041..5041 src has 310..5041) v3, discarding
  166. 2013-03-31 16:43:10.629280 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 42 <= 82968 0xd1eea80 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5041) v4, discarding
  167. 2013-03-31 16:43:11.445966 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 43 <= 82968 0x31db200 osd_map(5042..5042 src has 310..5042) v3, discarding
  168. 2013-03-31 16:43:11.446276 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 44 <= 82968 0x384a700 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5042) v4, discarding
  169. 2013-03-31 16:43:11.449167 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 45 <= 82968 0x384a700 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5042) v4, discarding
  170. 2013-03-31 16:43:24.629310 7fbe6dd1a700 0 log [WRN] : 5 slow requests, 5 included below; oldest blocked for > 30.825902 secs
  171. 2013-03-31 16:43:24.629317 7fbe6dd1a700 0 log [WRN] : slow request 30.825902 seconds old, received at 2013-03-31 16:42:53.803352: osd_op(client.7203.0:5883335 rbd_data.1e892ae8944a.00000000000097a0 [write 1576960~4096] 17.db072217 RETRY) v4 currently reached pg
  172. 2013-03-31 16:43:24.629322 7fbe6dd1a700 0 log [WRN] : slow request 30.311943 seconds old, received at 2013-03-31 16:42:54.317311: osd_op(client.7197.0:6616167 rbd_data.1c1a2ae8944a.00000000000088ac [write 3739648~8192] 17.ac5abd97 RETRY) v4 currently reached pg
  173. 2013-03-31 16:43:24.629326 7fbe6dd1a700 0 log [WRN] : slow request 30.311802 seconds old, received at 2013-03-31 16:42:54.317452: osd_op(client.7197.0:6616204 rbd_data.1c1a2ae8944a.000000000000893b [write 3866624~8192] 17.71a58617 RETRY) v4 currently reached pg
  174. 2013-03-31 16:43:24.629330 7fbe6dd1a700 0 log [WRN] : slow request 30.014494 seconds old, received at 2013-03-31 16:42:54.614760: osd_op(client.6912.0:123985196 4470.1_data/videoshots/339/339888-11.jpg [getxattrs,stat,read 0~524288] 10.7b1d23b9 RETRY) v4 currently reached pg
  175. 2013-03-31 16:43:24.629334 7fbe6dd1a700 0 log [WRN] : slow request 30.014312 seconds old, received at 2013-03-31 16:42:54.614942: osd_op(client.6912.0:123985481 4470.1_data/videoshots/6529/6529059-11.jpg [getxattrs,stat,read 0~524288] 10.9f102bb9 RETRY) v4 currently reached pg
  176. 2013-03-31 16:43:25.629486 7fbe6dd1a700 0 log [WRN] : 6 slow requests, 1 included below; oldest blocked for > 31.826104 secs
  177. 2013-03-31 16:43:25.629495 7fbe6dd1a700 0 log [WRN] : slow request 30.954455 seconds old, received at 2013-03-31 16:42:54.675001: osd_op(client.7197.0:6616337 rbd_data.1c1a2ae8944a.0000000000003257 [write 1466368~24576] 17.c38ed697) v4 currently reached pg
  178. 2013-03-31 16:43:27.084472 7fbe5fcfe700 0 log [INF] : 17.47 deep-scrub ok
  179. 2013-03-31 16:43:29.588625 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 46 <= 82968 0x31db200 osd_map(5043..5043 src has 310..5043) v3, discarding
  180. 2013-03-31 16:43:29.588883 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 47 <= 82968 0x3f12e00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5043) v4, discarding
  181. 2013-03-31 16:43:30.377951 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 48 <= 82968 0x31db200 osd_map(5044..5044 src has 310..5044) v3, discarding
  182. 2013-03-31 16:43:30.378421 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 49 <= 82968 0x94558c0 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5044) v4, discarding
  183. 2013-03-31 16:43:30.392476 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 50 <= 82968 0x94558c0 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5044) v4, discarding
  184. 2013-03-31 16:43:31.385751 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 51 <= 82968 0x31db200 osd_map(5045..5045 src has 310..5045) v3, discarding
  185. 2013-03-31 16:43:31.385889 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 52 <= 82968 0x9dfb1c0 pg_notify(0.28(3),1.27(3) epoch 5045) v4, discarding
  186. 2013-03-31 16:43:31.401801 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 53 <= 82968 0x9dfb1c0 pg_notify(2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5045) v4, discarding
  187. 2013-03-31 16:43:54.000362 7fbe6dd1a700 0 log [WRN] : 6 slow requests, 1 included below; oldest blocked for > 60.196984 secs
  188. 2013-03-31 16:43:54.000371 7fbe6dd1a700 0 log [WRN] : slow request 60.196984 seconds old, received at 2013-03-31 16:42:53.803352: osd_op(client.7203.0:5883335 rbd_data.1e892ae8944a.00000000000097a0 [write 1576960~4096] 17.db072217 RETRY) v4 currently reached pg
  189. 2013-03-31 16:43:55.000710 7fbe6dd1a700 0 log [WRN] : 6 slow requests, 5 included below; oldest blocked for > 61.197300 secs
  190. 2013-03-31 16:43:55.000718 7fbe6dd1a700 0 log [WRN] : slow request 60.683341 seconds old, received at 2013-03-31 16:42:54.317311: osd_op(client.7197.0:6616167 rbd_data.1c1a2ae8944a.00000000000088ac [write 3739648~8192] 17.ac5abd97 RETRY) v4 currently reached pg
  191. 2013-03-31 16:43:55.000723 7fbe6dd1a700 0 log [WRN] : slow request 60.683200 seconds old, received at 2013-03-31 16:42:54.317452: osd_op(client.7197.0:6616204 rbd_data.1c1a2ae8944a.000000000000893b [write 3866624~8192] 17.71a58617 RETRY) v4 currently reached pg
  192. 2013-03-31 16:43:55.000727 7fbe6dd1a700 0 log [WRN] : slow request 60.385892 seconds old, received at 2013-03-31 16:42:54.614760: osd_op(client.6912.0:123985196 4470.1_data/videoshots/339/339888-11.jpg [getxattrs,stat,read 0~524288] 10.7b1d23b9 RETRY) v4 currently reached pg
  193. 2013-03-31 16:43:55.000733 7fbe6dd1a700 0 log [WRN] : slow request 60.385710 seconds old, received at 2013-03-31 16:42:54.614942: osd_op(client.6912.0:123985481 4470.1_data/videoshots/6529/6529059-11.jpg [getxattrs,stat,read 0~524288] 10.9f102bb9 RETRY) v4 currently reached pg
  194. 2013-03-31 16:43:55.000736 7fbe6dd1a700 0 log [WRN] : slow request 60.325651 seconds old, received at 2013-03-31 16:42:54.675001: osd_op(client.7197.0:6616337 rbd_data.1c1a2ae8944a.0000000000003257 [write 1466368~24576] 17.c38ed697) v4 currently reached pg
  195. 2013-03-31 16:44:07.770087 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 54 <= 82968 0x701e000 osd_map(5046..5046 src has 310..5046) v3, discarding
  196. 2013-03-31 16:44:07.770359 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 55 <= 82968 0x486fa80 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3) epoch 5046) v4, discarding
  197. 2013-03-31 16:44:07.814043 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 56 <= 82968 0x486fa80 pg_notify(12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5046) v4, discarding
  198. 2013-03-31 16:44:08.741258 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 57 <= 82968 0x701e000 osd_map(5047..5047 src has 310..5047) v3, discarding
  199. 2013-03-31 16:44:08.741353 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 58 <= 82968 0xd66c540 pg_notify(2.26(3),11.1d(3) epoch 5047) v4, discarding
  200. 2013-03-31 16:44:08.743302 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 59 <= 82968 0xd66c540 pg_notify(13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5047) v4, discarding
  201. 2013-03-31 16:44:08.807604 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 60 <= 82968 0xd66c540 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5047) v4, discarding
  202. 2013-03-31 16:44:08.813047 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 61 <= 82968 0xd66c540 pg_notify(0.28(3),1.27(3) epoch 5047) v4, discarding
  203. 2013-03-31 16:44:09.003535 7fbe6dd1a700 0 log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 75.200145 secs
  204. 2013-03-31 16:44:09.003544 7fbe6dd1a700 0 log [WRN] : slow request 30.873763 seconds old, received at 2013-03-31 16:43:38.129734: osd_op(client.7227.0:7218713 rbd_data.1e8f2ae8944a.000000000000723f [write 2719744~4096] 17.f4510917) v4 currently reached pg
  205. 2013-03-31 16:44:09.821837 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 62 <= 82968 0xb9b0e00 osd_map(5048..5048 src has 310..5048) v3, discarding
  206. 2013-03-31 16:44:09.822154 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 63 <= 82968 0x5e72e00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5048) v4, discarding
  207. 2013-03-31 16:44:14.389871 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 64 <= 82968 0xd1c0a00 osd_map(5049..5049 src has 310..5049) v3, discarding
  208. 2013-03-31 16:44:14.390024 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 65 <= 82968 0x2ffee00 pg_notify(16.18(3),17.17(3),10.3b9(5) epoch 5049) v4, discarding
  209. 2013-03-31 16:44:14.403053 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 66 <= 82968 0x2ffee00 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3) epoch 5049) v4, discarding
  210. 2013-03-31 16:44:15.307650 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 67 <= 82968 0xd1c0a00 osd_map(5050..5050 src has 310..5050) v3, discarding
  211. 2013-03-31 16:44:15.307787 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 68 <= 82968 0x38e8000 pg_notify(1.27(3) epoch 5050) v4, discarding
  212. 2013-03-31 16:44:15.308568 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 69 <= 82968 0x38e8000 pg_notify(0.28(3) epoch 5050) v4, discarding
  213. 2013-03-31 16:44:15.382531 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 70 <= 82968 0x8820540 pg_notify(2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5050) v4, discarding
  214. 2013-03-31 16:44:32.456011 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 71 <= 82968 0xd469e00 osd_map(5051..5051 src has 310..5051) v3, discarding
  215. 2013-03-31 16:44:32.456197 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 72 <= 82968 0x6c9fc40 pg_notify(1.27(3) epoch 5051) v4, discarding
  216. 2013-03-31 16:44:32.458348 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 73 <= 82968 0x6c9fc40 pg_notify(0.28(3) epoch 5051) v4, discarding
  217. 2013-03-31 16:44:32.496631 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 74 <= 82968 0x6c9fc40 pg_notify(2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5051) v4, discarding
  218. 2013-03-31 16:44:33.603362 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 75 <= 82968 0xd469c00 osd_map(5052..5052 src has 310..5052) v3, discarding
  219. 2013-03-31 16:44:33.603619 7fbe5cff4700 0 -- 172.22.24.14:6830/7800 >> 172.22.24.15:6807/8954 pipe(0xb364500 sd=73 :49674 s=2 pgs=9323 cs=5 l=0).reader got old message 76 <= 82968 0x83f2a80 pg_notify(0.28(3),1.27(3),2.26(3),11.1d(3),13.1b(3),15.19(3),10.1e(3),14.1a(3),12.1c(3),16.18(3),17.17(3),10.3b9(5) epoch 5052) v4, discarding
  220. 2013-03-31 16:44:38.463905 7fbe6dd1a700 0 log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 104.660528 secs
  221. 2013-03-31 16:44:38.463916 7fbe6dd1a700 0 log [WRN] : slow request 60.334146 seconds old, received at 2013-03-31 16:43:38.129734: osd_op(client.7227.0:7218713 rbd_data.1e8f2ae8944a.000000000000723f [write 2719744~4096] 17.f4510917) v4 currently reached pg
  222. 2013-03-31 16:44:54.524531 7fbe6dd1a700 0 log [WRN] : 7 slow requests, 3 included below; oldest blocked for > 120.721134 secs
  223. 2013-03-31 16:44:54.524539 7fbe6dd1a700 0 log [WRN] : slow request 120.721134 seconds old, received at 2013-03-31 16:42:53.803352: osd_op(client.7203.0:5883335 rbd_data.1e892ae8944a.00000000000097a0 [write 1576960~4096] 17.db072217 RETRY) v4 currently reached pg
  224. 2013-03-31 16:44:54.524543 7fbe6dd1a700 0 log [WRN] : slow request 120.207175 seconds old, received at 2013-03-31 16:42:54.317311: osd_op(client.7197.0:6616167 rbd_data.1c1a2ae8944a.00000000000088ac [write 3739648~8192] 17.ac5abd97 RETRY) v4 currently reached pg
  225. 2013-03-31 16:44:54.524547 7fbe6dd1a700 0 log [WRN] : slow request 120.207034 seconds old, received at 2013-03-31 16:42:54.317452: osd_op(client.7197.0:6616204 rbd_data.1c1a2ae8944a.000000000000893b [write 3866624~8192] 17.71a58617 RETRY) v4 currently reached pg
  226. 2013-03-31 16:44:55.524854 7fbe6dd1a700 0 log [WRN] : 7 slow requests, 3 included below; oldest blocked for > 121.721452 secs
  227. 2013-03-31 16:44:55.524861 7fbe6dd1a700 0 log [WRN] : slow request 120.910044 seconds old, received at 2013-03-31 16:42:54.614760: osd_op(client.6912.0:123985196 4470.1_data/videoshots/339/339888-11.jpg [getxattrs,stat,read 0~524288] 10.7b1d23b9 RETRY) v4 currently reached pg
  228. 2013-03-31 16:44:55.524866 7fbe6dd1a700 0 log [WRN] : slow request 120.909862 seconds old, received at 2013-03-31 16:42:54.614942: osd_op(client.6912.0:123985481 4470.1_data/videoshots/6529/6529059-11.jpg [getxattrs,stat,read 0~524288] 10.9f102bb9 RETRY) v4 currently reached pg
  229. 2013-03-31 16:44:55.524870 7fbe6dd1a700 0 log [WRN] : slow request 120.849803 seconds old, received at 2013-03-31 16:42:54.675001: osd_op(client.7197.0:6616337 rbd_data.1c1a2ae8944a.0000000000003257 [write 1466368~24576] 17.c38ed697) v4 currently reached pg
  230. 2013-03-31 16:45:39.099662 7fbe6dd1a700 0 log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 165.296284 secs
  231. 2013-03-31 16:45:39.099667 7fbe6dd1a700 0 log [WRN] : slow request 120.969902 seconds old, received at 2013-03-31 16:43:38.129734: osd_op(client.7227.0:7218713 rbd_data.1e8f2ae8944a.000000000000723f [write 2719744~4096] 17.f4510917) v4 currently reached pg
  232. 2013-03-31 16:46:47.812273 7fbe6dd1a700 0 log [WRN] : 8 slow requests, 1 included below; oldest blocked for > 234.008881 secs
  233. 2013-03-31 16:46:47.812305 7fbe6dd1a700 0 log [WRN] : slow request 30.566556 seconds old, received at 2013-03-31 16:46:17.245677: osd_op(client.6912.0:123992084 4470.1_data/videoshots/914/914234-11.jpg [getxattrs,stat,read 0~524288] 10.be47ebb9) v4 currently reached pg
  234. 2013-03-31 16:46:54.057331 7fbe6dd1a700 0 log [WRN] : 8 slow requests, 1 included below; oldest blocked for > 240.253948 secs
  235. 2013-03-31 16:46:54.057340 7fbe6dd1a700 0 log [WRN] : slow request 240.253948 seconds old, received at 2013-03-31 16:42:53.803352: osd_op(client.7203.0:5883335 rbd_data.1e892ae8944a.00000000000097a0 [write 1576960~4096] 17.db072217 RETRY) v4 currently reached pg
  236. 2013-03-31 16:46:55.057490 7fbe6dd1a700 0 log [WRN] : 8 slow requests, 5 included below; oldest blocked for > 241.254081 secs
  237. 2013-03-31 16:46:55.057500 7fbe6dd1a700 0 log [WRN] : slow request 240.740122 seconds old, received at 2013-03-31 16:42:54.317311: osd_op(client.7197.0:6616167 rbd_data.1c1a2ae8944a.00000000000088ac [write 3739648~8192] 17.ac5abd97 RETRY) v4 currently reached pg
  238. 2013-03-31 16:46:55.057506 7fbe6dd1a700 0 log [WRN] : slow request 240.739981 seconds old, received at 2013-03-31 16:42:54.317452: osd_op(client.7197.0:6616204 rbd_data.1c1a2ae8944a.000000000000893b [write 3866624~8192] 17.71a58617 RETRY) v4 currently reached pg
  239. 2013-03-31 16:46:55.057510 7fbe6dd1a700 0 log [WRN] : slow request 240.442673 seconds old, received at 2013-03-31 16:42:54.614760: osd_op(client.6912.0:123985196 4470.1_data/videoshots/339/339888-11.jpg [getxattrs,stat,read 0~524288] 10.7b1d23b9 RETRY) v4 currently reached pg
  240. 2013-03-31 16:46:55.057513 7fbe6dd1a700 0 log [WRN] : slow request 240.442491 seconds old, received at 2013-03-31 16:42:54.614942: osd_op(client.6912.0:123985481 4470.1_data/videoshots/6529/6529059-11.jpg [getxattrs,stat,read 0~524288] 10.9f102bb9 RETRY) v4 currently reached pg
  241. 2013-03-31 16:46:55.057517 7fbe6dd1a700 0 log [WRN] : slow request 240.382432 seconds old, received at 2013-03-31 16:42:54.675001: osd_op(client.7197.0:6616337 rbd_data.1c1a2ae8944a.0000000000003257 [write 1466368~24576] 17.c38ed697) v4 currently reached pg
  242. 2013-03-31 16:47:14.264471 7fbe6e51b700 -1 osd.38 5052 *** Got signal Terminated ***
  243. 2013-03-31 16:47:14.264514 7fbe6e51b700 -1 osd.38 5052 pausing thread pools
  244. 2013-03-31 16:47:14.264521 7fbe6e51b700 -1 osd.38 5052 flushing io
  245. 2013-03-31 16:47:14.269750 7fbe6e51b700 -1 osd.38 5052 removing pid file
  246. 2013-03-31 16:47:14.269775 7fbe6e51b700 -1 osd.38 5052 exit
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!