API tools faq
paste
Advertisement
Guest User

flow

a guest
Aug 15th, 2014
57
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.08 KB | None | 0 0
raw download clone embed print report
  1. 2014-08-14T04:33:14.250068 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:33:14.250068+01:00 info: (/Stage[main]/Osnailyfacter::Cluster_ha/Exec[wait-for-haproxy-nova-backend]) Evaluated in 0.07 seconds
  2. 2014-08-14T04:33:15.139562 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:15.139562+01:00 alert: Server nova-metadata-api/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  3. 2014-08-14T04:33:16.205775 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.205775+01:00 alert: Server horizon/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  4. 2014-08-14T04:33:16.272730 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.272730+01:00 alert: Server horizon/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  5. 2014-08-14T04:33:16.405034 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.405034+01:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  6. 2014-08-14T04:33:16.472277 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.472277+01:00 alert: Server keystone-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  7. 2014-08-14T04:33:16.605669 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.605669+01:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  8. 2014-08-14T04:33:16.672802 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.672802+01:00 alert: Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  9. 2014-08-14T04:33:16.806192 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.806192+01:00 alert: Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  10. 2014-08-14T04:33:16.872304 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:16.872304+01:00 alert: Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  11. 2014-08-14T04:33:17.006564 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.006564+01:00 alert: Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  12. 2014-08-14T04:33:17.072837 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.072837+01:00 alert: Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  13. 2014-08-14T04:33:17.205300 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.205300+01:00 alert: Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  14. 2014-08-14T04:33:17.274093 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.274093+01:00 alert: Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  15. 2014-08-14T04:33:17.275358 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:17.275358+01:00 emerg: proxy nova-metadata-api has no server available!
  16. 2014-08-14T04:33:18.005752 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.005752+01:00 alert: Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  17. 2014-08-14T04:33:18.073574 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:18.073574+01:00 alert: Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  18. 2014-08-14T04:33:18.421727 node-2 ./remote/node-2.test.domain.local/CRON.log:2014-08-14T04:33:18.421727+01:00 info: pam_unix(cron:session): session closed for user root
  19. 2014-08-14T04:33:19.671615 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:19.671615+01:00 alert: Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 4000ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  20. 2014-08-14T04:33:20.205298 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:20.205298+01:00 alert: Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2899ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  21. 2014-08-14T04:33:21.753065 node-1 ./remote/node-1.test.domain.local/nova-conductor.log:2014-08-14T04:33:21.753065+01:00 debug: 2014-08-14 03:33:21.754 5860 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py:132
  22. 2014-08-14T04:33:22.472774 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:22.472774+01:00 alert: Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  23. 2014-08-14T04:33:22.676784 node-1 ./remote/node-1.test.domain.local/keystone-all.log:2014-08-14T04:33:22.676784+01:00 debug: 2014-08-14 03:33:22.677 6278 DEBUG keystone.common.kvs.core [-] KVS lock acquired for: usertokens-c363c631c43c4061906dcd1995398523 acquire /usr/lib/python2.7/dist-packages/keystone/common/kvs/core.py:376
  24. 2014-08-14T04:33:25.605725 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:33:25.605725+01:00 alert: Backup Server mysqld/node-2 is DOWN, reason: Layer4 timeout, check duration: 10001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  25. ...
  26. 2014-08-14T04:34:22.729037 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:34:22.729037+01:00 err: (/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.16.128-10.108.16.254]) Could not evaluate: Oops - not sure what happened: 751: unexpected token at '<html><body><h1>504 Gateway Time-out</h1>
  27. ...
  28. 2014-08-14T04:59:26.969424 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:26.969424+01:00 alert: Server nova-metadata-api/nod
  29. e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
  30. e, 0 requeued, 0 remaining in queue.
  31. 2014-08-14T04:59:27.969965 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:27.969965+01:00 alert: Server heat-api/node-1 is DO
  32. WN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requ
  33. eued, 0 remaining in queue.
  34. 2014-08-14T04:59:28.303341 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.303341+01:00 alert: Server horizon/node-2 is DOW
  35. N, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  36. 2014-08-14T04:59:28.351573 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:28.351573+01:00 alert: Server horizon/node-3 is DOW
  37. N, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  38. 2014-08-14T04:59:30.728760 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:30.728760+01:00 alert: Server nova-metadata-api/nod
  39. e-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions activ
  40. e, 0 requeued, 0 remaining in queue.
  41. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server horizon/node-2 is DOW
  42. N, reason: Layer4 timeout, check duration: 2228ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  43. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server horizon/node-3 is DOW
  44. N, reason: Layer4 timeout, check duration: 2180ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  45. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server keystone-1/node-2 is
  46. DOWN, reason: Layer4 timeout, check duration: 2084ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  47. 2014-08-14T04:59:31.729924 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:31.729924+01:00 alert: Server keystone-1/node-3 is
  48. DOWN, reason: Layer4 timeout, check duration: 2036ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  49. 2014-08-14T04:59:33.779265 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.779265+01:00 alert: Server keystone-2/node-2 is
  50. DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  51. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  52. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-1/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  53. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  54. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-2/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  55. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-api-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  56. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-metadata-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  57. 2014-08-14T04:59:33.780295 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780295+01:00 alert: Server nova-metadata-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  58. 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert: Server cinder-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  59. 2014-08-14T04:59:33.780381 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:33.780381+01:00 alert: Server cinder-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2028ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  60. 2014-08-14T04:59:35.691132 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.691132+01:00 alert: Backup Server mysqld/node-3 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  61. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Backup Server mysqld/node-2 is DOWN, reason: Layer4 connection problem, info: "No route to host", check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  62. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server heat-api/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  63. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server heat-api/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  64. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server nova-novncproxy/node-2 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  65. 2014-08-14T04:59:35.692311 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:35.692311+01:00 alert: Server nova-novncproxy/node-3 is DOWN, reason: Layer4 timeout, check duration: 2942ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  66. 2014-08-14T04:59:39.721140 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.721140+01:00 alert: Backup Server rabbitmq/node-2 is DOWN, reason: Layer4 timeout, check duration: 5916ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  67. 2014-08-14T04:59:39.722198 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:59:39.722198+01:00 alert: Backup Server rabbitmq/node-3 is DOWN, reason: Layer4 timeout, check duration: 5796ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!