craig-manzi1

Issues with Trove

May 8th, 2020
51
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 5.71 KB | None | 0 0
  1. # Issues with Trove
  2.  
  3. (Openstack = Rocky / Trove = 2.16.0)
  4.  
  5. ## Some background
  6. We have an existing Openstack which was deployed using Kolla/Kayobe. I have managed to add Trove and it is appearing in the Horizon dashboard.
  7. Created a test MySQL guest image using the trovestack integration script.
  8. Added this image to glance then created the datastore using that image.
  9. At first, when trying to build my instance it would just hang at "building" in the dashboard. Once checking the logs I could see it was sitting at the login prompt and had its IP address but was unable to gain access, via SSH or at the console.
  10. I decided to mounted the original image created from the integration script, using guestfish, and was able to create a new user and added in my SSH keys, then created a new instance with the edited image.
  11. I can now gain access to this instance via SSH, however, it is now dropping into an "Error" state on the Horizon Database dashboard and it looks like Trove is unable to communicate with the instance.
  12.  
  13. ## Troubleshooting
  14. After digging around our setup and it looks like we are missing the `trove-guestagent.conf` file, both on the instance and in the host config files.
  15.  
  16. I cloned the Trove files from GitHub and found a trove-guestagent.conf.sample file.
  17. I have found a similar-looking file on the actual taskmanager docker container called `/etc/trove/trove.conf`[1] but I would have expected more in it, I think ...., it has some ports defined but all URLs are 0.0.0.0.
  18.  
  19. The syslogs and the cloud-init logs[2] in the trove instance are saying the guest agent is failing to start, there is a critical error just before it tries to load the `/usr/local/bin/guest-agent` file (this only has 4 lines in it, would you expect more? then there are errors at almost every related python file it tries to run after.
  20.  
  21. When trying to run `/opt/guest-agent-venv/bin/trove-guestagent` manually on the instance here is what I get back:
  22. ```
  23. 2020-05-06 09:17:50.261 11276 CRITICAL root [-] Unhandled error: RuntimeError: Manager class not registered for datastore manager None
  24. 2020-05-06 09:17:50.261 11276 ERROR root Traceback (most recent call last):
  25. 2020-05-06 09:17:50.261 11276 ERROR root File "/opt/guest-agent-venv/bin/trove-guestagent", line 8, in <module>
  26. 2020-05-06 09:17:50.261 11276 ERROR root sys.exit(main())
  27. 2020-05-06 09:17:50.261 11276 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/cmd/guest.py", line 46, in main
  28. 2020-05-06 09:17:50.261 11276 ERROR root raise RuntimeError(msg)
  29. 2020-05-06 09:17:50.261 11276 ERROR root RuntimeError: Manager class not registered for datastore manager None
  30. 2020-05-06 09:17:50.261 11276 ERROR root
  31. ```
  32.  
  33. Then running it again with the (/usr/local/bin/guest-agent) config file defined you get:
  34. ```
  35. root@manzi-test-0405:~# /opt/guest-agent-venv/bin/trove-guestagent --config-file /etc/trove/conf.d/guest_info.conf
  36. 2020-05-06 09:21:23.530 11296 INFO trove.guestagent.module.driver_manager [-] Initializing module driver manager.
  37. 2020-05-06 09:21:23.538 11296 INFO trove.guestagent.module.driver_manager [-] Loading Module driver: ping
  38. 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loading Module driver: new_relic_license
  39. 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loaded module driver: ping
  40. 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loaded module driver: new_relic_license
  41. 2020-05-06 09:21:23.542 11296 CRITICAL root [-] Unhandled error: TypeError: sequence item 0: expected str instance, NoneType found
  42. 2020-05-06 09:21:23.542 11296 ERROR root Traceback (most recent call last):
  43. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/bin/trove-guestagent", line 8, in <module>
  44. 2020-05-06 09:21:23.542 11296 ERROR root sys.exit(main())
  45. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/cmd/guest.py", line 64, in main
  46. 2020-05-06 09:21:23.542 11296 ERROR root rpc_api_version=guest_api.API.API_LATEST_VERSION)
  47. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/common/rpc/service.py", line 49, in __init__
  48. 2020-05-06 09:21:23.542 11296 ERROR root self.manager_impl = profiler.trace_cls("rpc")(_manager)
  49. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/osprofiler/profiler.py", line 243, in decorator
  50. 2020-05-06 09:21:23.542 11296 ERROR root for attr_name, attr in inspect.getmembers(cls):
  51. 2020-05-06 09:21:23.542 11296 ERROR root File "/usr/lib/python3.5/inspect.py", line 304, in getmembers
  52. 2020-05-06 09:21:23.542 11296 ERROR root value = getattr(object, key)
  53. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/guestagent/datastore/manager.py", line 175, in guestagent_log_defs
  54. 2020-05-06 09:21:23.542 11296 ERROR root guestagent_log = guestagent_utils.build_file_path(log_dir, log_file)
  55. 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/guestagent/common/guestagent_utils.py", line 103, in build_file_path
  56. 2020-05-06 09:21:23.542 11296 ERROR root file_name = os.extsep.join([base_name] + list(extensions))
  57. 2020-05-06 09:21:23.542 11296 ERROR root TypeError: sequence item 0: expected str instance, NoneType found
  58. 2020-05-06 09:21:23.542 11296 ERROR root
  59. ```
  60.  
  61. Which is exactly what is happening in the syslogs.
  62.  
  63. Have you ever seen anything like this?
  64. Would you recommend copying over the trove.conf file from the taskmanager docker to the instance and see what happens?
  65.  
  66.  
  67. [1] (trove.conf) https://pastebin.com/DTywJ6xS
  68. [2] (Trove Logs) https://pastebin.com/RFPNCtee
Add Comment
Please, Sign In to add comment