ceph osd log

First sign of trouble:

2014-11-19 21:12:39.836171 7f7238f87700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f722f774700' had timed out after 15
..
2014-11-19 21:12:51.967429 7f72471af700  0 log_channel(default) log [WRN] : 18 slow requests, 18 included below; oldest blocked for > 30.808688 secs
2014-11-19 21:12:51.967445 7f72471af700  0 log_channel(default) log [WRN] : slow request 30.801559 seconds old, received at 2014-11-19 21:12:21.165078: osd_op(mds.0.3:336319 100000d1a4f.00000000 [create 0~0,setxattr parent (171)] 1.9889ccf1 ondisk+write+known_if_redirected e3573) currently waiting for subops from 2,3
..

Some repeats of this, number of slow requests increasing, then

2014-11-19 21:12:59.212046 7f72255b6700  0 -- 158.38.213.72:6806/4218 >> 158.38.213.73:6802/31405 pipe(0x29841b80 sd=47 :56301 s=2 pgs=32 cs=13 l=0 c=0x281c5a20).fault with nothing to send, going to standby

(4 different destinations)

..
2014-11-19 21:13:01.378591 7f723cf8f700  0 log_channel(default) log [WRN] : map e3575 wrongly marked me down
..
2014-11-19 21:13:29.187245 7f7228961700  0 -- 158.38.213.72:6810/1004218 >> 158.38.213.73:6802/31405 pipe(0x2da69b80 sd=33 :57720 s=2 pgs=34 cs=1 l=0 c=0x21c58160).fault, initiating reconnect

(A lot of repetitions of slightly different varitations of this one. 7-800 per second)

..
2014-11-19 21:14:57.813206 7f724b9f7700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f722f774700' had suicide timed out after 150
..
2014-11-19 21:14:57.912241 7f724b9f7700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7f724b9f7700 time 2014-11-19 21:14:57.814035
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0xc870b2]
 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2eb) [0xbcdb1b]
 3: (ceph::HeartbeatMap::is_healthy()+0xc6) [0xbce336]
 4: (ceph::HeartbeatMap::check_touch_file()+0x28) [0xbce9b8]
 5: (CephContextServiceThread::entry()+0x13d) [0xc9af1d]
 6: (()+0x6b50) [0x7f724e86cb50]
 7: (clone()+0x6d) [0x7f724d2887bd]