Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 2014-08-14T04:33:14.250068 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:33:14.250068+01:00 info: (/Stage[main]/Osnailyfacter::Cluster_ha/Exec[wait-for-haproxy-nova-backend]) Evaluated in 0.07 seconds
- 2014-08-14T04:33:15.139562 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:15.139562+01:00 alert: Server nova-metadata-api/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.205775 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.205775+01:00 alert: Server horizon/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.272730 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.272730+01:00 alert: Server horizon/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.405034 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.405034+01:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.472277 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.472277+01:00 alert: Server keystone-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.605669 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.605669+01:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.672802 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.672802+01:00 alert: Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.806192 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.806192+01:00 alert: Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:16.872304 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.872304+01:00 alert: Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:17.006564 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.006564+01:00 alert: Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:17.072837 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.072837+01:00 alert: Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:17.205300 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.205300+01:00 alert: Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:17.274093 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.274093+01:00 alert: Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:17.275358 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.275358+01:00 emerg: proxy nova-metadata-api has no server available!
- 2014-08-14T04:33:18.005752 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.005752+01:00 alert: Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:18.073574 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.073574+01:00 alert: Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:18.421727 node-2 ./remote/node-2.test.domain.local/CRON.log:2014-08-14T04:33:18.421727+01:00 info: pam_unix(cron:session): session closed for user root
- 2014-08-14T04:33:19.671615 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:19.671615+01:00 alert: Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 4000ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:20.205298 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:20.205298+01:00 alert: Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2899ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:21.753065 node-1 ./remote/node-1.test.domain.local/nova-conductor.log:2014-08-14T04:33:21.753065+01:00 debug: 2014-08-14 03:33:21.754 5860 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py:132
- 2014-08-14T04:33:22.472774 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:22.472774+01:00 alert: Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:33:22.676784 node-1 ./remote/node-1.test.domain.local/keystone-all.log:2014-08-14T04:33:22.676784+01:00 debug: 2014-08-14 03:33:22.677 6278 DEBUG keystone.common.kvs.core [-] KVS lock acquired for: usertokens-c363c631c43c4061906dcd1995398523 acquire /usr/lib/python2.7/dist-packages/keystone/common/kvs/core.py:376
- 2014-08-14T04:33:25.605725 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:25.605725+01:00 alert: Backup Server mysqld/node-2 is DOWN, reason: Layer4 timeout, check duration: 10001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- ...
- 2014-08-14T04:34:22.729037 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:34:22.729037+01:00 err: (/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.16.128-10.108.16.254]) Could not evaluate: Oops - not sure what happened: 751: unexpected token at '<html><body><h1>504 Gateway Time-out</h1>
- ...
- 2014-08-14T04:59:26.969424 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:26.969424+01:00 alert: Server nova-metadata-api/nod
- e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
- e, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:27.969965 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:27.969965+01:00 alert: Server heat-api/node-1 is DO
- WN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requ
- eued, 0 remaining in queue.
- 2014-08-14T04:59:28.303341 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.303341+01:00 alert: Server horizon/node-2 is DOW
- N, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:28.351573 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.351573+01:00 alert: Server horizon/node-3 is DOW
- N, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:30.728760 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:30.728760+01:00 alert: Server nova-metadata-api/nod
- e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
- e, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server horizon/node-2 is DOW
- N, reason: Layer4 timeout, check duration: 2228ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server horizon/node-3 is DOW
- N, reason: Layer4 timeout, check duration: 2180ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server keystone-1/node-2 is
- DOWN, reason: Layer4 timeout, check duration: 2084ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server keystone-1/node-3 is
- DOWN, reason: Layer4 timeout, check duration: 2036ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.779265 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.779265+01:00 alert: Server keystone-2/node-2 is
- DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert: Server cinder-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert: Server cinder-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.691132 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.691132+01:00 alert: Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Backup Server mysqld/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server heat-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server heat-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:39.721140 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.721140+01:00 alert: Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 timeout, check duration: 5916ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
- 2014-08-14T04:59:39.722198 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.722198+01:00 alert: Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5796ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement