API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Mar 4th, 2013
81
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 64.81 KB | None | 0 0
raw download clone embed print report
  1.  
  2. 2013-03-04 17:57:58.479104 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  3. 2013-03-04 17:57:58.519853 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  4. 2013-03-04 17:57:58.655271 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  5. 2013-03-04 17:57:59.579766 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  6. 2013-03-04 17:57:59.614020 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  7. 2013-03-04 17:57:59.871087 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  8. 2013-03-04 17:57:59.999376 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  9. 2013-03-04 17:58:00.274177 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  10. 2013-03-04 17:58:00.401840 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  11. 2013-03-04 17:58:00.958833 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  12. 2013-03-04 17:58:00.971578 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  13. 2013-03-04 17:58:01.280707 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  14. 2013-03-04 17:58:01.301242 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  15. 2013-03-04 17:58:01.374910 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  16. 2013-03-04 17:58:01.420434 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  17. 2013-03-04 17:58:02.755979 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  18. 2013-03-04 17:58:02.981460 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  19. 2013-03-04 17:58:03.001875 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  20. 2013-03-04 17:58:03.500617 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  21. 2013-03-04 17:58:03.714680 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  22. 2013-03-04 17:58:03.872289 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  23. 2013-03-04 17:58:04.702644 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  24. 2013-03-04 17:58:04.921186 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  25. 2013-03-04 17:58:05.056591 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  26. 2013-03-04 17:58:05.200958 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  27. 2013-03-04 17:58:05.300181 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  28. 2013-03-04 17:58:05.401961 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  29. 2013-03-04 17:58:05.415524 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  30. 2013-03-04 17:58:06.516222 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  31. 2013-03-04 17:58:06.675749 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  32. 2013-03-04 17:58:06.859439 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  33. 2013-03-04 17:58:06.901621 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  34. 2013-03-04 17:58:07.682077 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  35. 2013-03-04 17:58:07.957146 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  36. 2013-03-04 17:58:08.803222 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  37. 2013-03-04 17:58:09.021913 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  38. 2013-03-04 17:58:09.173120 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  39. 2013-03-04 17:58:09.202184 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  40. 2013-03-04 17:58:09.416820 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  41. 2013-03-04 17:58:09.702906 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  42. 2013-03-04 17:58:10.402142 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  43. 2013-03-04 17:58:10.503817 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  44. 2013-03-04 17:58:10.517489 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  45. 2013-03-04 17:58:10.776581 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  46. 2013-03-04 17:58:11.201127 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  47. 2013-03-04 17:58:11.604441 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  48. 2013-03-04 17:58:12.160037 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  49. 2013-03-04 17:58:12.382786 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  50. 2013-03-04 17:58:12.818205 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  51. 2013-03-04 17:58:13.258029 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  52. 2013-03-04 17:58:13.273912 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  53. 2013-03-04 17:58:13.677284 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  54. 2013-03-04 17:58:13.918988 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  55. 2013-03-04 17:58:14.322701 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  56. 2013-03-04 17:58:14.403768 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  57. 2013-03-04 17:58:14.460565 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  58. 2013-03-04 17:58:14.505131 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  59. 2013-03-04 17:58:14.958702 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  60. 2013-03-04 17:58:15.402294 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  61. 2013-03-04 17:58:15.605914 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  62. 2013-03-04 17:58:16.174546 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  63. 2013-03-04 17:58:16.501901 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  64. 2013-03-04 17:58:16.578034 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  65. 2013-03-04 17:58:16.819809 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  66. 2013-03-04 17:58:17.223572 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  67. 2013-03-04 17:58:17.304593 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  68. 2013-03-04 17:58:17.574190 7f47c6645700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6810/5881 pipe(0x183e1500 sd=31 :39569 s=2 pgs=973 cs=1 l=0).fault with nothing to send, going to standby
  69. 2013-03-04 17:58:17.577686 7f47c6140700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6821/7645 pipe(0x17b8dc80 sd=32 :32774 s=2 pgs=27 cs=1 l=0).fault with nothing to send, going to standby
  70. 2013-03-04 17:58:17.586372 7f47c5b3a700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6819/7391 pipe(0x16675280 sd=33 :36928 s=2 pgs=98 cs=1 l=0).fault with nothing to send, going to standby
  71. 2013-03-04 17:58:17.609582 7f47c6342700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6808/5139 pipe(0x17b8da00 sd=36 :43901 s=2 pgs=471 cs=1 l=0).fault with nothing to send, going to standby
  72. 2013-03-04 17:58:17.609581 7f47c805f700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6801/6655 pipe(0x17b8d280 sd=34 :42084 s=2 pgs=171 cs=1 l=0).fault with nothing to send, going to standby
  73. 2013-03-04 17:58:17.609665 7f47c8665700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6831/6916 pipe(0x18874500 sd=39 :55908 s=2 pgs=207 cs=1 l=0).fault with nothing to send, going to standby
  74. 2013-03-04 17:58:18.283422 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  75. 2013-03-04 17:58:18.578411 7f47c6746700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6805/7138 pipe(0x17b8d000 sd=46 :6803 s=2 pgs=1418 cs=1 l=0).fault with nothing to send, going to standby
  76. 2013-03-04 17:58:18.676883 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:57:58.676882)
  77. 2013-03-04 17:58:18.704493 7f47c7756700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6823/6399 pipe(0x14df4780 sd=52 :6803 s=2 pgs=1187 cs=1 l=0).fault with nothing to send, going to standby
  78. 2013-03-04 17:58:19.384047 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  79. 2013-03-04 17:58:19.874963 7f47c7352700 0 -- 192.168.21.134:6803/5586 >> 192.168.21.134:6815/6141 pipe(0x14df4000 sd=53 :6803 s=2 pgs=1276 cs=1 l=0).fault with nothing to send, going to standby
  80. 2013-03-04 17:58:20.402451 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  81. 2013-03-04 17:58:21.577609 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:01.577609)
  82. 2013-03-04 17:58:22.284781 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  83. 2013-03-04 17:58:22.785440 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  84. 2013-03-04 17:58:25.402582 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  85. 2013-03-04 17:58:25.686247 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  86. 2013-03-04 17:58:27.478256 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:07.478255)
  87. 2013-03-04 17:58:27.478270 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:07.478255)
  88. 2013-03-04 17:58:27.478281 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:07.478255)
  89. 2013-03-04 17:58:30.387158 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  90. 2013-03-04 17:58:30.402716 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  91. 2013-03-04 17:58:32.778997 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:12.778996)
  92. 2013-03-04 17:58:32.779010 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:12.778996)
  93. 2013-03-04 17:58:32.779016 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:12.778996)
  94. 2013-03-04 17:58:32.779022 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:12.778996)
  95. 2013-03-04 17:58:32.779027 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:12.778996)
  96. 2013-03-04 17:58:35.079823 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:15.079822)
  97. 2013-03-04 17:58:35.079836 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:15.079822)
  98. 2013-03-04 17:58:35.079842 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:15.079822)
  99. 2013-03-04 17:58:35.079848 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:15.079822)
  100. 2013-03-04 17:58:35.079853 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:15.079822)
  101. 2013-03-04 17:58:35.402871 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  102. 2013-03-04 17:58:35.688086 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  103. 2013-03-04 17:58:37.980626 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:17.980625)
  104. 2013-03-04 17:58:37.980639 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:17.980625)
  105. 2013-03-04 17:58:37.980646 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:17.980625)
  106. 2013-03-04 17:58:37.980651 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:17.980625)
  107. 2013-03-04 17:58:37.980657 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:17.980625)
  108. 2013-03-04 17:58:40.389228 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  109. 2013-03-04 17:58:40.403005 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  110. 2013-03-04 17:58:40.881365 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:20.881364)
  111. 2013-03-04 17:58:40.881373 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:20.881364)
  112. 2013-03-04 17:58:40.881378 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:20.881364)
  113. 2013-03-04 17:58:40.881384 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:20.881364)
  114. 2013-03-04 17:58:40.881390 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:20.881364)
  115. 2013-03-04 17:58:44.490298 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  116. 2013-03-04 17:58:45.403173 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  117. 2013-03-04 17:58:46.782176 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:26.782176)
  118. 2013-03-04 17:58:46.782206 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:26.782176)
  119. 2013-03-04 17:58:46.782274 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:26.782176)
  120. 2013-03-04 17:58:46.782280 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:26.782176)
  121. 2013-03-04 17:58:46.782285 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:26.782176)
  122. 2013-03-04 17:58:47.883058 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:27.883057)
  123. 2013-03-04 17:58:47.883084 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:27.883057)
  124. 2013-03-04 17:58:47.883091 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:27.883057)
  125. 2013-03-04 17:58:47.883097 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:27.883057)
  126. 2013-03-04 17:58:47.883102 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:27.883057)
  127. 2013-03-04 17:58:49.791113 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  128. 2013-03-04 17:58:50.291809 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  129. 2013-03-04 17:58:50.403285 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  130. 2013-03-04 17:58:52.584044 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:32.584044)
  131. 2013-03-04 17:58:52.584082 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:32.584044)
  132. 2013-03-04 17:58:52.584090 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:32.584044)
  133. 2013-03-04 17:58:52.584096 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:32.584044)
  134. 2013-03-04 17:58:52.584102 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:32.584044)
  135. 2013-03-04 17:58:55.403396 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  136. 2013-03-04 17:58:55.592555 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  137. 2013-03-04 17:58:56.084850 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:36.084849)
  138. 2013-03-04 17:58:56.084857 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:36.084849)
  139. 2013-03-04 17:58:56.084863 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:36.084849)
  140. 2013-03-04 17:58:56.084869 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:36.084849)
  141. 2013-03-04 17:58:56.084875 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:36.084849)
  142. 2013-03-04 17:58:59.693327 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  143. 2013-03-04 17:59:00.403563 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  144. 2013-03-04 17:59:00.785604 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:40.785604)
  145. 2013-03-04 17:59:00.785619 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:40.785604)
  146. 2013-03-04 17:59:00.785625 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:40.785604)
  147. 2013-03-04 17:59:00.785629 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:40.785604)
  148. 2013-03-04 17:59:00.785638 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:40.785604)
  149. 2013-03-04 17:59:00.794225 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  150. 2013-03-04 17:59:03.695227 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  151. 2013-03-04 17:59:04.886238 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:44.886237)
  152. 2013-03-04 17:59:04.886246 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:44.886237)
  153. 2013-03-04 17:59:04.886252 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:44.886237)
  154. 2013-03-04 17:59:04.886258 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:44.886237)
  155. 2013-03-04 17:59:04.886265 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:44.886237)
  156. 2013-03-04 17:59:05.403667 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  157. 2013-03-04 17:59:06.586874 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:46.586873)
  158. 2013-03-04 17:59:06.586882 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:46.586873)
  159. 2013-03-04 17:59:06.586887 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:46.586873)
  160. 2013-03-04 17:59:06.586893 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:46.586873)
  161. 2013-03-04 17:59:06.586904 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:46.586873)
  162. 2013-03-04 17:59:07.196272 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  163. 2013-03-04 17:59:08.897219 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  164. 2013-03-04 17:59:09.487380 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:49.487379)
  165. 2013-03-04 17:59:09.487394 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:49.487379)
  166. 2013-03-04 17:59:09.487399 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:49.487379)
  167. 2013-03-04 17:59:09.487405 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:49.487379)
  168. 2013-03-04 17:59:09.487410 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:49.487379)
  169. 2013-03-04 17:59:10.403813 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  170. 2013-03-04 17:59:11.188238 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:51.188238)
  171. 2013-03-04 17:59:11.188254 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:51.188238)
  172. 2013-03-04 17:59:11.188260 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:51.188238)
  173. 2013-03-04 17:59:11.188266 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:51.188238)
  174. 2013-03-04 17:59:11.188272 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:51.188238)
  175. 2013-03-04 17:59:12.398116 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  176. 2013-03-04 17:59:14.689106 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:54.689105)
  177. 2013-03-04 17:59:14.689121 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:54.689105)
  178. 2013-03-04 17:59:14.689128 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:54.689105)
  179. 2013-03-04 17:59:14.689133 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:54.689105)
  180. 2013-03-04 17:59:14.689139 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:54.689105)
  181. 2013-03-04 17:59:15.403949 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  182. 2013-03-04 17:59:18.189898 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:58.189897)
  183. 2013-03-04 17:59:18.189913 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:58.189897)
  184. 2013-03-04 17:59:18.189919 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:58.189897)
  185. 2013-03-04 17:59:18.189925 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:58.189897)
  186. 2013-03-04 17:59:18.189930 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:58.189897)
  187. 2013-03-04 17:59:18.298912 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  188. 2013-03-04 17:59:18.690661 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:58:58.690660)
  189. 2013-03-04 17:59:18.690674 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:58.690660)
  190. 2013-03-04 17:59:18.690680 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:58.690660)
  191. 2013-03-04 17:59:18.690686 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:58:58.690660)
  192. 2013-03-04 17:59:18.690691 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:58:58.690660)
  193. 2013-03-04 17:59:20.404080 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  194. 2013-03-04 17:59:20.599720 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  195. 2013-03-04 17:59:22.191543 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:02.191542)
  196. 2013-03-04 17:59:22.191555 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:02.191542)
  197. 2013-03-04 17:59:22.191561 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:02.191542)
  198. 2013-03-04 17:59:22.191566 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:02.191542)
  199. 2013-03-04 17:59:22.191571 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:02.191542)
  200. 2013-03-04 17:59:23.892276 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:03.892275)
  201. 2013-03-04 17:59:23.892286 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:03.892275)
  202. 2013-03-04 17:59:23.892292 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:03.892275)
  203. 2013-03-04 17:59:23.892298 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:03.892275)
  204. 2013-03-04 17:59:23.892304 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:03.892275)
  205. 2013-03-04 17:59:25.404214 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  206. 2013-03-04 17:59:25.900346 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  207. 2013-03-04 17:59:29.793055 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:09.793054)
  208. 2013-03-04 17:59:29.793063 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:09.793054)
  209. 2013-03-04 17:59:29.793068 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:09.793054)
  210. 2013-03-04 17:59:29.793073 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:09.793054)
  211. 2013-03-04 17:59:29.793079 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:09.793054)
  212. 2013-03-04 17:59:30.404348 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  213. 2013-03-04 17:59:31.201094 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  214. 2013-03-04 17:59:34.101848 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  215. 2013-03-04 17:59:34.493957 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:14.493956)
  216. 2013-03-04 17:59:34.493967 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:14.493956)
  217. 2013-03-04 17:59:34.493973 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:14.493956)
  218. 2013-03-04 17:59:34.493979 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:14.493956)
  219. 2013-03-04 17:59:34.493985 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:14.493956)
  220. 2013-03-04 17:59:35.404488 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  221. 2013-03-04 17:59:36.402592 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  222. 2013-03-04 17:59:39.194763 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:19.194763)
  223. 2013-03-04 17:59:39.194776 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:19.194763)
  224. 2013-03-04 17:59:39.194782 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:19.194763)
  225. 2013-03-04 17:59:39.194788 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:19.194763)
  226. 2013-03-04 17:59:39.194794 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:19.194763)
  227. 2013-03-04 17:59:39.903388 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  228. 2013-03-04 17:59:40.404105 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  229. 2013-03-04 17:59:40.404577 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  230. 2013-03-04 17:59:43.295681 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:23.295681)
  231. 2013-03-04 17:59:43.295694 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:23.295681)
  232. 2013-03-04 17:59:43.295700 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:23.295681)
  233. 2013-03-04 17:59:43.295707 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:23.295681)
  234. 2013-03-04 17:59:43.295719 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:23.295681)
  235. 2013-03-04 17:59:43.904846 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  236. 2013-03-04 17:59:45.404709 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  237. 2013-03-04 17:59:48.596460 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:28.596459)
  238. 2013-03-04 17:59:48.596472 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:28.596459)
  239. 2013-03-04 17:59:48.596478 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:28.596459)
  240. 2013-03-04 17:59:48.596483 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:28.596459)
  241. 2013-03-04 17:59:48.596489 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:28.596459)
  242. 2013-03-04 17:59:49.697362 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:29.697361)
  243. 2013-03-04 17:59:49.697376 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:29.697361)
  244. 2013-03-04 17:59:49.697382 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:29.697361)
  245. 2013-03-04 17:59:49.697390 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:29.697361)
  246. 2013-03-04 17:59:49.697396 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:29.697361)
  247. 2013-03-04 17:59:49.805520 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  248. 2013-03-04 17:59:50.404829 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  249. 2013-03-04 17:59:51.398137 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:31.398137)
  250. 2013-03-04 17:59:51.398150 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:31.398137)
  251. 2013-03-04 17:59:51.398155 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:31.398137)
  252. 2013-03-04 17:59:51.398161 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:31.398137)
  253. 2013-03-04 17:59:51.398167 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:31.398137)
  254. 2013-03-04 17:59:51.899044 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:31.899043)
  255. 2013-03-04 17:59:51.899060 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:31.899043)
  256. 2013-03-04 17:59:51.899067 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:31.899043)
  257. 2013-03-04 17:59:51.899072 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:31.899043)
  258. 2013-03-04 17:59:51.899079 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:31.899043)
  259. 2013-03-04 17:59:52.106248 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  260. 2013-03-04 17:59:54.199862 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:34.199862)
  261. 2013-03-04 17:59:54.199874 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:34.199862)
  262. 2013-03-04 17:59:54.199880 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:34.199862)
  263. 2013-03-04 17:59:54.199886 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:34.199862)
  264. 2013-03-04 17:59:54.199925 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:34.199862)
  265. 2013-03-04 17:59:55.006896 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  266. 2013-03-04 17:59:55.404971 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  267. 2013-03-04 17:59:55.900607 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:35.900606)
  268. 2013-03-04 17:59:55.900622 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:35.900606)
  269. 2013-03-04 17:59:55.900627 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:35.900606)
  270. 2013-03-04 17:59:55.900633 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:35.900606)
  271. 2013-03-04 17:59:55.900639 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:35.900606)
  272. 2013-03-04 17:59:59.107551 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  273. 2013-03-04 18:00:00.405116 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  274. 2013-03-04 18:00:00.808230 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  275. 2013-03-04 18:00:01.201426 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:41.201425)
  276. 2013-03-04 18:00:01.201434 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:41.201425)
  277. 2013-03-04 18:00:01.201439 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:41.201425)
  278. 2013-03-04 18:00:01.201445 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:41.201425)
  279. 2013-03-04 18:00:01.201450 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:41.201425)
  280. 2013-03-04 18:00:04.308967 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  281. 2013-03-04 18:00:05.405263 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  282. 2013-03-04 18:00:05.902363 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:45.902363)
  283. 2013-03-04 18:00:05.902376 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:45.902363)
  284. 2013-03-04 18:00:05.902382 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:45.902363)
  285. 2013-03-04 18:00:05.902388 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:45.902363)
  286. 2013-03-04 18:00:05.902395 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:45.902363)
  287. 2013-03-04 18:00:06.609720 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  288. 2013-03-04 18:00:08.310400 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  289. 2013-03-04 18:00:08.803299 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:48.803298)
  290. 2013-03-04 18:00:08.803311 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:48.803298)
  291. 2013-03-04 18:00:08.803317 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:48.803298)
  292. 2013-03-04 18:00:08.803322 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:48.803298)
  293. 2013-03-04 18:00:08.803340 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:48.803298)
  294. 2013-03-04 18:00:08.811186 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  295. 2013-03-04 18:00:09.312028 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  296. 2013-03-04 18:00:09.812890 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  297. 2013-03-04 18:00:10.405447 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  298. 2013-03-04 18:00:12.904121 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:52.904121)
  299. 2013-03-04 18:00:12.904132 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:52.904121)
  300. 2013-03-04 18:00:12.904138 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:52.904121)
  301. 2013-03-04 18:00:12.904144 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:52.904121)
  302. 2013-03-04 18:00:12.904150 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:52.904121)
  303. 2013-03-04 18:00:15.204905 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 17:59:55.204904)
  304. 2013-03-04 18:00:15.204920 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:55.204904)
  305. 2013-03-04 18:00:15.204926 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:55.204904)
  306. 2013-03-04 18:00:15.204933 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 17:59:55.204904)
  307. 2013-03-04 18:00:15.204939 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 17:59:55.204904)
  308. 2013-03-04 18:00:15.405620 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  309. 2013-03-04 18:00:15.713857 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  310. 2013-03-04 18:00:20.405776 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  311. 2013-03-04 18:00:20.505966 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:00.505963)
  312. 2013-03-04 18:00:20.505984 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:00.505963)
  313. 2013-03-04 18:00:20.505989 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:00.505963)
  314. 2013-03-04 18:00:20.505995 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:00.505963)
  315. 2013-03-04 18:00:20.506000 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:00.505963)
  316. 2013-03-04 18:00:21.007005 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:01.007004)
  317. 2013-03-04 18:00:21.007020 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:01.007004)
  318. 2013-03-04 18:00:21.007026 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:01.007004)
  319. 2013-03-04 18:00:21.007032 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:01.007004)
  320. 2013-03-04 18:00:21.007037 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:01.007004)
  321. 2013-03-04 18:00:21.014662 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  322. 2013-03-04 18:00:21.515397 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  323. 2013-03-04 18:00:23.907937 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:03.907936)
  324. 2013-03-04 18:00:23.907952 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:03.907936)
  325. 2013-03-04 18:00:23.907959 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:03.907936)
  326. 2013-03-04 18:00:23.908039 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:03.907936)
  327. 2013-03-04 18:00:23.908046 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:03.907936)
  328. 2013-03-04 18:00:25.405880 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  329. 2013-03-04 18:00:27.416012 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  330. 2013-03-04 18:00:29.808964 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:09.808963)
  331. 2013-03-04 18:00:29.808985 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:09.808963)
  332. 2013-03-04 18:00:29.808992 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:09.808963)
  333. 2013-03-04 18:00:29.808998 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:09.808963)
  334. 2013-03-04 18:00:29.809005 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:09.808963)
  335. 2013-03-04 18:00:30.406057 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  336. 2013-03-04 18:00:31.516800 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  337. 2013-03-04 18:00:34.509814 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:14.509813)
  338. 2013-03-04 18:00:34.509827 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:14.509813)
  339. 2013-03-04 18:00:34.509833 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:14.509813)
  340. 2013-03-04 18:00:34.509839 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:14.509813)
  341. 2013-03-04 18:00:34.509845 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:14.509813)
  342. 2013-03-04 18:00:35.406163 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  343. 2013-03-04 18:00:36.217542 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  344. 2013-03-04 18:00:37.918120 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  345. 2013-03-04 18:00:39.018832 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  346. 2013-03-04 18:00:40.406314 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  347. 2013-03-04 18:00:40.410653 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:20.410652)
  348. 2013-03-04 18:00:40.410662 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:20.410652)
  349. 2013-03-04 18:00:40.410668 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:20.410652)
  350. 2013-03-04 18:00:40.410672 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:20.410652)
  351. 2013-03-04 18:00:40.410676 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:20.410652)
  352. 2013-03-04 18:00:41.919739 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  353. 2013-03-04 18:00:44.820583 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  354. 2013-03-04 18:00:45.111593 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:25.111592)
  355. 2013-03-04 18:00:45.111602 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:25.111592)
  356. 2013-03-04 18:00:45.111608 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:25.111592)
  357. 2013-03-04 18:00:45.111614 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:25.111592)
  358. 2013-03-04 18:00:45.111620 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:25.111592)
  359. 2013-03-04 18:00:45.406467 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  360. 2013-03-04 18:00:48.612344 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:28.612344)
  361. 2013-03-04 18:00:48.612352 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:28.612344)
  362. 2013-03-04 18:00:48.612358 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:28.612344)
  363. 2013-03-04 18:00:48.612364 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:28.612344)
  364. 2013-03-04 18:00:48.612369 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:28.612344)
  365. 2013-03-04 18:00:49.713179 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:29.713178)
  366. 2013-03-04 18:00:49.713194 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:29.713178)
  367. 2013-03-04 18:00:49.713201 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:29.713178)
  368. 2013-03-04 18:00:49.713267 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:29.713178)
  369. 2013-03-04 18:00:49.713274 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:29.713178)
  370. 2013-03-04 18:00:50.121326 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  371. 2013-03-04 18:00:50.406624 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  372. 2013-03-04 18:00:52.014221 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:32.014220)
  373. 2013-03-04 18:00:52.014235 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:32.014220)
  374. 2013-03-04 18:00:52.014241 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:32.014220)
  375. 2013-03-04 18:00:52.014247 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:32.014220)
  376. 2013-03-04 18:00:52.014252 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:32.014220)
  377. 2013-03-04 18:00:53.022061 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  378. 2013-03-04 18:00:55.406806 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  379. 2013-03-04 18:00:56.715118 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:36.715117)
  380. 2013-03-04 18:00:56.715142 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:36.715117)
  381. 2013-03-04 18:00:56.715149 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:36.715117)
  382. 2013-03-04 18:00:56.715155 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:36.715117)
  383. 2013-03-04 18:00:56.715161 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:36.715117)
  384. 2013-03-04 18:00:58.922902 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  385. 2013-03-04 18:01:00.406992 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  386. 2013-03-04 18:01:02.015994 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:42.015993)
  387. 2013-03-04 18:01:02.016006 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:42.015993)
  388. 2013-03-04 18:01:02.016013 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:42.015993)
  389. 2013-03-04 18:01:02.016018 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:42.015993)
  390. 2013-03-04 18:01:02.016024 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:42.015993)
  391. 2013-03-04 18:01:02.516954 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:42.516953)
  392. 2013-03-04 18:01:02.516974 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:42.516953)
  393. 2013-03-04 18:01:02.516981 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:42.516953)
  394. 2013-03-04 18:01:02.516986 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:42.516953)
  395. 2013-03-04 18:01:02.516991 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:42.516953)
  396. 2013-03-04 18:01:04.223576 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  397. 2013-03-04 18:01:05.407088 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  398. 2013-03-04 18:01:07.124314 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  399. 2013-03-04 18:01:08.417804 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:48.417804)
  400. 2013-03-04 18:01:08.417829 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:48.417804)
  401. 2013-03-04 18:01:08.417835 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:48.417804)
  402. 2013-03-04 18:01:08.417840 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:48.417804)
  403. 2013-03-04 18:01:08.417845 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:48.417804)
  404. 2013-03-04 18:01:10.025028 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  405. 2013-03-04 18:01:10.407230 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  406. 2013-03-04 18:01:11.125835 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  407. 2013-03-04 18:01:11.918602 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:51.918601)
  408. 2013-03-04 18:01:11.918613 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:51.918601)
  409. 2013-03-04 18:01:11.918619 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:51.918601)
  410. 2013-03-04 18:01:11.918625 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:51.918601)
  411. 2013-03-04 18:01:11.918630 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:51.918601)
  412. 2013-03-04 18:01:13.619418 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:53.619417)
  413. 2013-03-04 18:01:13.619436 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:53.619417)
  414. 2013-03-04 18:01:13.619444 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:53.619417)
  415. 2013-03-04 18:01:13.619450 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:53.619417)
  416. 2013-03-04 18:01:13.619456 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:53.619417)
  417. 2013-03-04 18:01:14.719895 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:54.719894)
  418. 2013-03-04 18:01:14.719903 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:54.719894)
  419. 2013-03-04 18:01:14.719908 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:54.719894)
  420. 2013-03-04 18:01:14.719913 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:54.719894)
  421. 2013-03-04 18:01:14.719919 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:54.719894)
  422. 2013-03-04 18:01:15.407403 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  423. 2013-03-04 18:01:17.020560 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:57.020559)
  424. 2013-03-04 18:01:17.020568 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:57.020559)
  425. 2013-03-04 18:01:17.020572 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:57.020559)
  426. 2013-03-04 18:01:17.020576 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:57.020559)
  427. 2013-03-04 18:01:17.020579 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:57.020559)
  428. 2013-03-04 18:01:17.026723 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  429. 2013-03-04 18:01:18.721405 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:00:58.721404)
  430. 2013-03-04 18:01:18.721416 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:58.721404)
  431. 2013-03-04 18:01:18.721422 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:58.721404)
  432. 2013-03-04 18:01:18.721428 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:00:58.721404)
  433. 2013-03-04 18:01:18.721434 7f47ca26c700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:00:58.721404)
  434. 2013-03-04 18:01:20.407540 7f47daa8d700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  435. 2013-03-04 18:01:21.127543 7f47cea75700 1 heartbeat_map is_healthy 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  436. 2013-03-04 18:01:22.868816 7f47cb26e700 1 heartbeat_map reset_timeout 'OSD::disk_tp thread 0x7f47cb26e700' had timed out after 60
  437. 2013-03-04 18:01:22.870965 7f47d928a700 -1 osd.10 3637 heartbeat_check: no reply from osd.1 since 2013-03-04 17:57:56.669740 (cutoff 2013-03-04 18:01:02.870964)
  438. 2013-03-04 18:01:22.871000 7f47d928a700 -1 osd.10 3637 heartbeat_check: no reply from osd.2 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:01:02.870964)
  439. 2013-03-04 18:01:22.871006 7f47d928a700 -1 osd.10 3637 heartbeat_check: no reply from osd.3 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:01:02.870964)
  440. 2013-03-04 18:01:22.871012 7f47d928a700 -1 osd.10 3637 heartbeat_check: no reply from osd.4 since 2013-03-04 17:58:08.074982 (cutoff 2013-03-04 18:01:02.870964)
  441. 2013-03-04 18:01:22.871018 7f47d928a700 -1 osd.10 3637 heartbeat_check: no reply from osd.7 since 2013-03-04 17:58:06.374135 (cutoff 2013-03-04 18:01:02.870964)
  442. 2013-03-04 18:01:22.874171 7f47d127a700 0 log [WRN] : map e3646 wrongly marked me down
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!