Pioneer: Savefile versioning and compatibility

From: John Bartholomew
To: Pioneer development discussion
Subject: Savefile versioning and compatibility
Date: Wednesday, October 10, 2012 1:24 AM

I've been thinking about this for a while. It's one of the tricky but
absolutely necessary items on the todo list to eventually get us out
of alpha.

Some things I think we should be aiming for:

- No more global save version; it's just not compatible with a system
that is continually in development.
- Structure metadata in the save file, with structure remapping to
support loading older (or newer) files. For a good example of a system
like this with files that have remained compatible (both forwards and
backwards) over a long and very active development life, see Blender.
- Coping with loading into a galaxy that doesn't exactly match the one
that the save was made from. This is a tough one, but can be done with
some care and pragmatism (more below).
- Coping with changes to available equipment, missions, etc. I would
suggest this is mostly deferred to Lua, which doesn't solve it, but at
least admits flexible solutions. Missions are already in Lua and there
is a pull request open to add some basic versioning to mission save
data. Equipment will be handled by Lua in the future, and that whole
system should be easier to make robust to version changes.
- Coping with changes to other content (available ships, etc). I don't
think there is much to do here except reset to some default ship if
the one in the save file doesn't exist.

Some notes on galaxy changes:
In the past I've thought that we could eventually lock down the galaxy
(when sysgen has been cleaned up), which would make things easier, but
given the chance of custom systems and adjustments to terrain
appearing as mods, and given how long it could take to get sysgen into
a state that we are happy to make permanent, I think we're better off
putting in the effort to cope with loading into a galaxy that is (a
little) different to the one the save was made in.

This needs some adjustments to the way some data is serialized to make
it cope better with changes. Eg, if close to a body, store altitude
rather than distance to frame centre (so if terrain height is
different on load then the ship won't be underground); if docked, just
store station path and landing pad number, so if the station is
positioned differently on load the ship will still load in a sensible
place. There is probably a bunch of stuff like this that can be done
to make the system a little more robust to changes, and it's not
hugely difficult, but it needs thought and care in a lot of places.

I think SystemPath, or at least the on-disk representation will need
to change. Possibly we can say system names must be unique within a
sector, and serialize SystemPath as <sector, system name, body name>.
Then of names can't be changed, so it's not a perfect solution. I'm
not sure what will be best; I don't think there are any perfect
solutions to this, we just have to pick a trade-off that we're happy
with.

Some changes to the galaxy can only be handled by resetting some state
on load, eg if the ship is in a system that doesn't exist at all then
it can be loaded into a nearby system, or loaded into an empty ghost
system at the coordinates of the saved system (think witch-space) to
give the pilot a chance to jump to the system of their choice.

There is probably more to say, but that's all I have time for right now.

John B