Pioneer: Savefile upgrades - a modest proposal

From: Robert Norris
To: [email protected]
Subject: Savefile upgrades - a modest proposal
Date: Wednesday, December 19, 2012 10:32 PM

We've known we have to tackle savefile upgrades (backward-compatibility)
for a while. For some reason I found myself thinking about it again
today, and got motivated enough to write some code on the way home.

Right now I'm concentrating more on the on-disk format rather than the
data within it. John B had some really good thoughts about how we might
change the data we store in a previous post[1], and I think we
absolutely have to do some of that, but right now I'm just thinking
about a format that can cope with changes in stored data assuming
everything else (positions, terrain, etc) doesn't change.

Now we certainly could make what we have upgradeable. Its "just" a small
matter of reading the old format, putting the data into a new format and
plugging any gaps with sane defaults. The trouble with the current
format is that its only structure is tied to the order its written and
read in. If you're off by even one byte, everything explodes. This is a
healthy source of bugs for anything that touches it, including
conversion code.

The format is also not particularly easy to extend. This makes life
difficult when you want to store arbitrary structured data in it, like
anything from Lua. As we put more and more functionality out to Lua,
this is going to become more of a problem. Really, the needs of the core
are minimal in comparison.

So, some rough requirements/desirables:

 - Order not significant.
 - Can sanely store Lua data types.
 - Can store floating-point numbers without loss.
 - Works cross-platform.

Here's my plan, based on some playing I'm doing on
robn/savefile-experiment.

 - JSON. The disk format is fairly unimportant, but JSON is nice to work
   with, familiar to many, human readable, has close to a 1:1 mapping to
   Lua, and is basically a hashtable (dictionary) format, so can handle
   unordered, nested data. I'm not particularly wedded to JSON but I'm
   already planning to bring in a JSON lib[2] for the RPC stuff I'm
   doing, so we might as well use it over yet another format. In
   particular the serialisation/deserialisation of Lua data is
   incredibly easy.

 - The native JSON number type is defined as arbitrary-precision.
   jsoncpp uses up to 16 fractional digits (the C format "%#.16g") but
   its still a base-10 format, so subject to loss. But, I think we can
   work around this by encoding doubles using the C99 %a format, for
   hexadecimal notation. To mark this in the file for the parser, we'll
   store a double (or float) as [ "d", "0x9.e0652141ef0dbf6p-3" ]
   (that's 1.23456789). Its still relatively human-readable (maybe
   human-decipherable is a better term).

 - The only really cross-platform concerns are float construction and
   endianness. Endianness is not a problem for JSON (not that we work
   anywhere that isn't Intel anyway), and as far as I can tell from a
   light skim of the relevant bits of the glibc code, %a should convert
   back and forth anywhere.

That's the format itself. The way I want to handle upgrades is to store
a numeric version in a metadata section, and then in the code, we
provide a number of conversion methods. Each one converts from version N
to version N+1. If we stick to our tradition that we do one version bump
per month, then each individual method should be fairly small, and we
can gradually remove older ones quite simply.

To do an upgrade of an older savefile, we just progressively run the
data through the necessary methods until we get it up to the latest
version.

So I think that will do for now. Like I said, initially this is only
about the data format, so no interface changes (at least on the Lua
side). I think the ability to load out-of-order will help with a lot of
stuff in the future, but that's for after this is done. Something I
think will be easy will be to not call the Lua loaders until the core
game state is established, so there won't be a need to do the
onGameStart post-load stuff that we do now. That's a big win right
there.

I'll be hacking on this over the next little while, in amongst all the
other junk I'm working on, so you'll see this in PR before long. But I'm
interested to hear any thoughts you have and especially anything I've
missed.

Cheers,
Rob.