API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jan 13th, 2016
176
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.21 KB | None | 0 0
raw download clone embed print report
  1. grep '070995e4-66f4-4acd-87c7-92815a489edd' ./remote/node-7.domain.tld/nova-compute.log
  2. ....
  3. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] Traceback (most recent call last):
  4. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1738, in _prep_block_device
  5. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] wait_func=self._await_block_device_map_created)
  6. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 476, in attach_block_devices
  7. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] map(_log_and_attach, block_device_mapping)
  8. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 474, in _log_and_attach
  9. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] bdm.attach(*attach_args, **attach_kwargs)
  10. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 383, in attach
  11. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] availability_zone=av_zone)
  12. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 379, in create
  13. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] item = client.volumes.create(size, **kwargs)
  14. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volumes.py", line 187, in create
  15. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] return self._create('/volumes', body, 'volume')
  16. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 181, in _create
  17. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] resp, body = self.api.client.post(url, body=body)
  18. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 104, in post
  19. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] return self._cs_request(url, 'POST', **kwargs)
  20. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 98, in _cs_request
  21. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] return self.request(url, method, **kwargs)
  22. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 91, in request
  23. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] raise exceptions.from_response(resp, body)
  24. 2016-01-12 09:43:03.155 21110 ERROR nova.compute.manager [instance: 070995e4-66f4-4acd-87c7-92815a489edd] ClientException: Gateway Time-out (HTTP 504)
  25.  
  26. haproxy logs have a lot of 504 answers
  27.  
  28. ogubanov@ogubanov-pc:~/Downloads/logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep -c ' 504 '
  29. 161
  30.  
  31. and found that haproxy (seems) reset PORT requests if it takes >= 60 second
  32.  
  33. ogubanov@ogubanov-pc:~/Downloads/logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep ' 504 ' | tail -n 3
  34. ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:50:37.451451+00:00 info: 192.168.0.35:49739 [12/Jan/2016:09:49:37.453] cinder-api cinder-api/node-20 0/0/0/-1/60001 504 194 - - sH-- 1848/24/24/24/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/20d52a22-efdd-4848-b82e-c9a347510adb/action HTTP/1.1"
  35. ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:51:06.319330+00:00 info: 192.168.0.23:47543 [12/Jan/2016:09:50:06.300] cinder-api cinder-api/node-20 0/0/0/-1/60011 504 194 - - sH-- 1862/23/23/23/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/4f453d0f-6602-476c-8330-1ef1caa91aa1/action HTTP/1.1"
  36. ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:51:06.330370+00:00 info: 192.168.0.36:41157 [12/Jan/2016:09:50:06.302] cinder-api cinder-api/node-20 0/0/0/-1/60021 504 194 - - sH-- 1861/22/22/22/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/f19c61b7-ebc8-40cc-af57-1840a59ae428/action HTTP/1.1"
  37.  
  38. you can compare that normal request takes less that 60 second
  39. find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep 'POST /v1/fcf23cb570d640ffb4c93d4dcca9c138'
  40.  
  41.  
  42. grep -B20 'Connection reset by peer' ./docker-logs/node-20.domain.tld/cinder-api.log
  43. 2016-01-12T09:51:16.451286+00:00 debug: 2016-01-12 09:51:16.454 19544 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 3a4f0e7389b14681bd45be6c691c8e2c __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:324
  44. 2016-01-12T09:51:16.457713+00:00 info: 2016-01-12 09:51:16.455 19544 INFO cinder.volume.api [req-8af95330-4f33-41e6-9123-0f8254fa0638 e637067843a745d9ae19d52c97137608 fcf23cb570d640ffb4c93d4dcca9c138 - - -] Terminate volume connection completed successfully.
  45. 2016-01-12T09:51:16.472114+00:00 debug: 2016-01-12 09:51:16.475 19543 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2d5619a7d4ef49b29addad1014928fed __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:324
  46. 2016-01-12T09:51:16.474389+00:00 info: 2016-01-12 09:51:16.476 19543 INFO cinder.volume.api [req-865c8d93-b099-40db-8263-eb14c3fd10dd e637067843a745d9ae19d52c97137608 fcf23cb570d640ffb4c93d4dcca9c138 - - -] Terminate volume connection completed successfully.
  47. 2016-01-12T09:51:16.505569+00:00 debug: 2016-01-12 09:51:16.509 19527 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 573c8daf7eb44ec58fed1ff4cef39274 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:324
  48. 2016-01-12T09:51:16.507635+00:00 info: 2016-01-12 09:51:16.510 19527 INFO cinder.volume.api [req-ac55b31c-50a9-4bf5-bcc1-cc411425fad2 e637067843a745d9ae19d52c97137608 fcf23cb570d640ffb4c93d4dcca9c138 - - -] Attach volume completed successfully.
  49. 2016-01-12T09:51:16.509457+00:00 info: 2016-01-12 09:51:16.511 19527 INFO cinder.api.openstack.wsgi [req-ac55b31c-50a9-4bf5-bcc1-cc411425fad2 e637067843a745d9ae19d52c97137608 fcf23cb570d640ffb4c93d4dcca9c138 - - -] http://192.168.0.2:8776/v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/4f453d0f-6602-476c-8330-1ef1caa91aa1/action returned with HTTP 202
  50. 2016-01-12T09:51:16.513772+00:00 info: 2016-01-12 09:51:16.513 19527 INFO eventlet.wsgi.server [req-ac55b31c-50a9-4bf5-bcc1-cc411425fad2 e637067843a745d9ae19d52c97137608 fcf23cb570d640ffb4c93d4dcca9c138 - - -] Traceback (most recent call last):
  51. File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 483, in handle_one_response
  52. write(b''.join(towrite))
  53. File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 426, in write
  54. _writelines(towrite)
  55. File "/usr/lib/python2.7/socket.py", line 334, in writelines
  56. self.flush()
  57. File "/usr/lib/python2.7/socket.py", line 303, in flush
  58. self._sock.sendall(view[write_offset:write_offset+buffer_size])
  59. File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 377, in sendall
  60. tail = self.send(data, flags)
  61. File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 359, in send
  62. total_sent += fd.send(data[total_sent:], flags)
  63. error: [Errno 104] Connection reset by peer
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!