daily pastebin goal
81%
SHARE
TWEET

flow

a guest Aug 15th, 2014 7 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. 2014-08-14T04:33:14.250068 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:33:14.250068+01:00 info:  (/Stage[main]/Osnailyfacter::Cluster_ha/Exec[wait-for-haproxy-nova-backend]) Evaluated in 0.07 seconds
  2. 2014-08-14T04:33:15.139562 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:15.139562+01:00 alert:  Server nova-metadata-api/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  3. 2014-08-14T04:33:16.205775 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.205775+01:00 alert:  Server horizon/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  4. 2014-08-14T04:33:16.272730 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.272730+01:00 alert:  Server horizon/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  5. 2014-08-14T04:33:16.405034 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.405034+01:00 alert:  Server keystone-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  6. 2014-08-14T04:33:16.472277 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.472277+01:00 alert:  Server keystone-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  7. 2014-08-14T04:33:16.605669 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.605669+01:00 alert:  Server keystone-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  8. 2014-08-14T04:33:16.672802 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.672802+01:00 alert:  Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  9. 2014-08-14T04:33:16.806192 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.806192+01:00 alert:  Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  10. 2014-08-14T04:33:16.872304 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.872304+01:00 alert:  Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  11. 2014-08-14T04:33:17.006564 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.006564+01:00 alert:  Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  12. 2014-08-14T04:33:17.072837 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.072837+01:00 alert:  Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  13. 2014-08-14T04:33:17.205300 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.205300+01:00 alert:  Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  14. 2014-08-14T04:33:17.274093 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.274093+01:00 alert:  Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  15. 2014-08-14T04:33:17.275358 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.275358+01:00 emerg:  proxy nova-metadata-api has no server available!
  16. 2014-08-14T04:33:18.005752 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.005752+01:00 alert:  Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  17. 2014-08-14T04:33:18.073574 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.073574+01:00 alert:  Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  18. 2014-08-14T04:33:18.421727 node-2 ./remote/node-2.test.domain.local/CRON.log:2014-08-14T04:33:18.421727+01:00 info: pam_unix(cron:session): session closed for user root
  19. 2014-08-14T04:33:19.671615 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:19.671615+01:00 alert:  Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 4000ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  20. 2014-08-14T04:33:20.205298 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:20.205298+01:00 alert:  Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2899ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  21. 2014-08-14T04:33:21.753065 node-1 ./remote/node-1.test.domain.local/nova-conductor.log:2014-08-14T04:33:21.753065+01:00 debug:  2014-08-14 03:33:21.754 5860 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py:132
  22. 2014-08-14T04:33:22.472774 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:22.472774+01:00 alert:  Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  23. 2014-08-14T04:33:22.676784 node-1 ./remote/node-1.test.domain.local/keystone-all.log:2014-08-14T04:33:22.676784+01:00 debug:  2014-08-14 03:33:22.677 6278 DEBUG keystone.common.kvs.core [-] KVS lock acquired for: usertokens-c363c631c43c4061906dcd1995398523 acquire /usr/lib/python2.7/dist-packages/keystone/common/kvs/core.py:376
  24. 2014-08-14T04:33:25.605725 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:25.605725+01:00 alert:  Backup Server mysqld/node-2 is DOWN, reason: Layer4 timeout, check duration: 10001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  25. ...
  26. 2014-08-14T04:34:22.729037 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:34:22.729037+01:00 err:  (/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.16.128-10.108.16.254]) Could not evaluate: Oops - not sure what happened: 751: unexpected token at '<html><body><h1>504 Gateway Time-out</h1>
  27. ...
  28. 2014-08-14T04:59:26.969424 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:26.969424+01:00 alert:  Server nova-metadata-api/nod
  29. e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
  30. e, 0 requeued, 0 remaining in queue.
  31. 2014-08-14T04:59:27.969965 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:27.969965+01:00 alert:  Server heat-api/node-1 is DO
  32. WN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requ
  33. eued, 0 remaining in queue.
  34. 2014-08-14T04:59:28.303341 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.303341+01:00 alert:  Server horizon/node-2 is DOW
  35. N, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  36. 2014-08-14T04:59:28.351573 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.351573+01:00 alert:  Server horizon/node-3 is DOW
  37. N, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  38. 2014-08-14T04:59:30.728760 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:30.728760+01:00 alert:  Server nova-metadata-api/nod
  39. e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
  40. e, 0 requeued, 0 remaining in queue.
  41. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert:  Server horizon/node-2 is DOW
  42. N, reason: Layer4 timeout, check duration: 2228ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  43. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert:  Server horizon/node-3 is DOW
  44. N, reason: Layer4 timeout, check duration: 2180ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  45. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert:  Server keystone-1/node-2 is
  46. DOWN, reason: Layer4 timeout, check duration: 2084ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  47. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert:  Server keystone-1/node-3 is
  48. DOWN, reason: Layer4 timeout, check duration: 2036ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  49. 2014-08-14T04:59:33.779265 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.779265+01:00 alert:  Server keystone-2/node-2 is
  50. DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  51. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  52. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  53. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  54. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  55. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  56. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  57. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert:  Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  58. 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert:  Server cinder-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  59. 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert:  Server cinder-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  60. 2014-08-14T04:59:35.691132 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.691132+01:00 alert:  Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  61. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert:  Backup Server mysqld/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  62. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert:  Server heat-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  63. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert:  Server heat-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  64. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert:  Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  65. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert:  Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  66. 2014-08-14T04:59:39.721140 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.721140+01:00 alert:  Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 timeout, check duration: 5916ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  67. 2014-08-14T04:59:39.722198 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.722198+01:00 alert:  Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5796ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top