joao@timesink:~/inktank/src/teuthology/virtualenv/bin$ ../../hammer.sh -a ~/inktank/issues/2480/config.yaml
+ [ -a = -a ]
+ shift
+ job=/home/joao/inktank/issues/2480/config.yaml
+ log=--archive /home/joao/inktank/issues/2480/config.yaml.out
+ test -e /home/joao/inktank/issues/2480/config.yaml
+ N=0
+ title
+ echo \[\033]0;hammer /home/joao/inktank/issues/2480/config.yaml 0 passes\007\]
\[\]
+ [ -n --archive /home/joao/inktank/issues/2480/config.yaml.out ]
+ [ -d /home/joao/inktank/issues/2480/config.yaml.out ]
+ rm -rf /home/joao/inktank/issues/2480/config.yaml.out
+ teuthology --archive /home/joao/inktank/issues/2480/config.yaml.out /home/joao/inktank/issues/2480/config.yaml
INFO:teuthology.run_tasks:Running task internal.save_config...
INFO:teuthology.task.internal:Saving configuration
INFO:teuthology.run_tasks:Running task internal.check_lock...
INFO:teuthology.task.internal:Checking locks...
INFO:teuthology.run_tasks:Running task internal.connect...
INFO:teuthology.task.internal:Opening connections...
INFO:teuthology.task.internal:roles: ubuntu@plana25.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2']
INFO:teuthology.task.internal:roles: ubuntu@plana48.front.sepia.ceph.com - ['mon.b', 'mds.a', 'osd.3', 'osd.4', 'osd.5']
INFO:teuthology.task.internal:roles: ubuntu@plana55.front.sepia.ceph.com - ['client.0']
INFO:teuthology.run_tasks:Running task internal.check_conflict...
INFO:teuthology.task.internal:Checking for old test directory...
INFO:teuthology.run_tasks:Running task internal.base...
INFO:teuthology.task.internal:Creating base directory...
INFO:teuthology.run_tasks:Running task internal.archive...
INFO:teuthology.task.internal:Creating archive directory...
INFO:teuthology.run_tasks:Running task internal.coredump...
INFO:teuthology.task.internal:Enabling coredump saving...
INFO:teuthology.orchestra.run.out:kernel.core_pattern = /tmp/cephtest/archive/coredump/%t.%p.core
INFO:teuthology.orchestra.run.out:kernel.core_pattern = /tmp/cephtest/archive/coredump/%t.%p.core
INFO:teuthology.orchestra.run.out:kernel.core_pattern = /tmp/cephtest/archive/coredump/%t.%p.core
INFO:teuthology.run_tasks:Running task internal.syslog...
INFO:teuthology.task.internal:Starting syslog monitoring...
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2697
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2492
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2491
INFO:teuthology.run_tasks:Running task internal.timer...
INFO:teuthology.task.internal:Starting timer...
INFO:teuthology.run_tasks:Running task mon_recovery...
INFO:teuthology.task.mon_recovery:mon ids = ['a', 'c', 'b']
INFO:teuthology.task.mon_recovery.ceph_manager:waiting for quorum size 3
INFO:teuthology.orchestra.run.err:bash: /tmp/cephtest/enable-coredump: No such file or directory
ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
  File "/home/joao/inktank/src/teuthology/teuthology/run_tasks.py", line 25, in run_tasks
    manager = _run_one_task(taskname, ctx=ctx, config=config)
  File "/home/joao/inktank/src/teuthology/teuthology/run_tasks.py", line 14, in _run_one_task
    return fn(**kwargs)
  File "/home/joao/inktank/src/teuthology/teuthology/task/mon_recovery.py", line 28, in task
    manager.wait_for_mon_quorum_size(len(mons))
  File "/home/joao/inktank/src/teuthology/teuthology/task/ceph_manager.py", line 403, in wait_for_mon_quorum_size
    while not len(self.get_mon_quorum()) == size:
  File "/home/joao/inktank/src/teuthology/teuthology/task/ceph_manager.py", line 395, in get_mon_quorum
    out = self.raw_cluster_cmd('quorum_status')
  File "/home/joao/inktank/src/teuthology/teuthology/task/ceph_manager.py", line 158, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/joao/inktank/src/teuthology/teuthology/orchestra/remote.py", line 40, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/joao/inktank/src/teuthology/teuthology/orchestra/run.py", line 257, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/joao/inktank/src/teuthology/teuthology/orchestra/run.py", line 253, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status)
CommandFailedError: Command failed with status 127: 'LD_LIBRARY_PRELOAD=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/ceph -k /tmp/cephtest/ceph.keyring -c /tmp/cephtest/ceph.conf --concise quorum_status'
INFO:teuthology.task.internal:Duration was 0.043440 seconds
INFO:teuthology.task.internal:Shutting down syslog monitoring...
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2708
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2502
INFO:teuthology.orchestra.run.out:rsyslog start/running, process 2501
INFO:teuthology.task.internal:Checking logs for errors...
INFO:teuthology.task.internal:Compressing syslogs...
INFO:teuthology.orchestra.run.out:kernel.core_pattern = core
INFO:teuthology.orchestra.run.out:kernel.core_pattern = core
INFO:teuthology.orchestra.run.out:kernel.core_pattern = core
INFO:teuthology.task.internal:Transferring archived files...
INFO:teuthology.task.internal:Removing archive directory...
INFO:teuthology.task.internal:Tidying up after the test...
+ echo /home/joao/inktank/issues/2480/config.yaml: 0 passes, then failure.
/home/joao/inktank/issues/2480/config.yaml: 0 passes, then failure.
joao@timesink:~/inktank/src/teuthology/virtualenv/bin$


---
config.yaml:


overrides:
  ceph:
    conf:
      osd:
        osd op complaint time: 120
      mon:
        debug mon: 20
        debug ms: 10
    fs: btrfs
    log-whitelist:
    - clocks not synchronized
    - slow request
    sha1: 73e29b658dfba231ea491178314ad419c8813f93
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
 [the targets]
tasks:
- mon_recovery: null