go into HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers and set homeworld.exe to
~ DWM8And16BitMitigation NT4SP5

Then attempt to start it.

Then go to:
HKEY_CLASSES_ROOT\VirtualStore\MACHINE\SOFTWARE\Wow6432Node\Sierra On-Line\Homeworld
and set glToSelect to opengl32.dll, set screendepth to 32, and set your resolution.


(unrelated) For visual quality, make sure to disable texture filtering in-game (enabled by default) and force anisotropic filtering in your video driver panel instead.