Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- # Issues with Trove
- (Openstack = Rocky / Trove = 2.16.0)
- ## Some background
- We have an existing Openstack which was deployed using Kolla/Kayobe. I have managed to add Trove and it is appearing in the Horizon dashboard.
- Created a test MySQL guest image using the trovestack integration script.
- Added this image to glance then created the datastore using that image.
- At first, when trying to build my instance it would just hang at "building" in the dashboard. Once checking the logs I could see it was sitting at the login prompt and had its IP address but was unable to gain access, via SSH or at the console.
- I decided to mounted the original image created from the integration script, using guestfish, and was able to create a new user and added in my SSH keys, then created a new instance with the edited image.
- I can now gain access to this instance via SSH, however, it is now dropping into an "Error" state on the Horizon Database dashboard and it looks like Trove is unable to communicate with the instance.
- ## Troubleshooting
- After digging around our setup and it looks like we are missing the `trove-guestagent.conf` file, both on the instance and in the host config files.
- I cloned the Trove files from GitHub and found a trove-guestagent.conf.sample file.
- I have found a similar-looking file on the actual taskmanager docker container called `/etc/trove/trove.conf`[1] but I would have expected more in it, I think ...., it has some ports defined but all URLs are 0.0.0.0.
- The syslogs and the cloud-init logs[2] in the trove instance are saying the guest agent is failing to start, there is a critical error just before it tries to load the `/usr/local/bin/guest-agent` file (this only has 4 lines in it, would you expect more? then there are errors at almost every related python file it tries to run after.
- When trying to run `/opt/guest-agent-venv/bin/trove-guestagent` manually on the instance here is what I get back:
- ```
- 2020-05-06 09:17:50.261 11276 CRITICAL root [-] Unhandled error: RuntimeError: Manager class not registered for datastore manager None
- 2020-05-06 09:17:50.261 11276 ERROR root Traceback (most recent call last):
- 2020-05-06 09:17:50.261 11276 ERROR root File "/opt/guest-agent-venv/bin/trove-guestagent", line 8, in <module>
- 2020-05-06 09:17:50.261 11276 ERROR root sys.exit(main())
- 2020-05-06 09:17:50.261 11276 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/cmd/guest.py", line 46, in main
- 2020-05-06 09:17:50.261 11276 ERROR root raise RuntimeError(msg)
- 2020-05-06 09:17:50.261 11276 ERROR root RuntimeError: Manager class not registered for datastore manager None
- 2020-05-06 09:17:50.261 11276 ERROR root
- ```
- Then running it again with the (/usr/local/bin/guest-agent) config file defined you get:
- ```
- root@manzi-test-0405:~# /opt/guest-agent-venv/bin/trove-guestagent --config-file /etc/trove/conf.d/guest_info.conf
- 2020-05-06 09:21:23.530 11296 INFO trove.guestagent.module.driver_manager [-] Initializing module driver manager.
- 2020-05-06 09:21:23.538 11296 INFO trove.guestagent.module.driver_manager [-] Loading Module driver: ping
- 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loading Module driver: new_relic_license
- 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loaded module driver: ping
- 2020-05-06 09:21:23.541 11296 INFO trove.guestagent.module.driver_manager [-] Loaded module driver: new_relic_license
- 2020-05-06 09:21:23.542 11296 CRITICAL root [-] Unhandled error: TypeError: sequence item 0: expected str instance, NoneType found
- 2020-05-06 09:21:23.542 11296 ERROR root Traceback (most recent call last):
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/bin/trove-guestagent", line 8, in <module>
- 2020-05-06 09:21:23.542 11296 ERROR root sys.exit(main())
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/cmd/guest.py", line 64, in main
- 2020-05-06 09:21:23.542 11296 ERROR root rpc_api_version=guest_api.API.API_LATEST_VERSION)
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/common/rpc/service.py", line 49, in __init__
- 2020-05-06 09:21:23.542 11296 ERROR root self.manager_impl = profiler.trace_cls("rpc")(_manager)
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/osprofiler/profiler.py", line 243, in decorator
- 2020-05-06 09:21:23.542 11296 ERROR root for attr_name, attr in inspect.getmembers(cls):
- 2020-05-06 09:21:23.542 11296 ERROR root File "/usr/lib/python3.5/inspect.py", line 304, in getmembers
- 2020-05-06 09:21:23.542 11296 ERROR root value = getattr(object, key)
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/guestagent/datastore/manager.py", line 175, in guestagent_log_defs
- 2020-05-06 09:21:23.542 11296 ERROR root guestagent_log = guestagent_utils.build_file_path(log_dir, log_file)
- 2020-05-06 09:21:23.542 11296 ERROR root File "/opt/guest-agent-venv/lib/python3.5/site-packages/trove/guestagent/common/guestagent_utils.py", line 103, in build_file_path
- 2020-05-06 09:21:23.542 11296 ERROR root file_name = os.extsep.join([base_name] + list(extensions))
- 2020-05-06 09:21:23.542 11296 ERROR root TypeError: sequence item 0: expected str instance, NoneType found
- 2020-05-06 09:21:23.542 11296 ERROR root
- ```
- Which is exactly what is happening in the syslogs.
- Have you ever seen anything like this?
- Would you recommend copying over the trove.conf file from the taskmanager docker to the instance and see what happens?
- [1] (trove.conf) https://pastebin.com/DTywJ6xS
- [2] (Trove Logs) https://pastebin.com/RFPNCtee
Add Comment
Please, Sign In to add comment