Overview
========

This is the storage server for the XXX website.

Production
----------

This buildout sets up a ZEO storage server. Make sure to run this buildout
with the same Python version, as you run the ZEO clients.

Currently this sets up a ZEO 3.9 server with one filestorage and a blobstorage.

To run this buildout you do::

  $ python2.4 bootstrap.py
  $ bin/buildout

This sets up the ZEO server, supervisord, log rotation and configures cron
entries to restart and do regular maintenance. The ZEO server is protected via
an authentication database.


Development
-----------

If you want to test this locally and do some development, you can use the
``devel.cfg`` buildout file::

  $ python2.4 bootstrap.py
  $ bin/buildout -c devel.cfg

This will avoid creating crontab entries or log rotation functionality.


Garbage collection
------------------

We don't do garbage collection as part of a pack anymore. Instead we use
explicit garbage collection via a different job::

  bin/multi-zodb-gc etc/zeo.conf

Garbage collection can take between one and two hours.


Packing
-------

We do regular packing every day, via::

  bin/zeopack

Packing without garbage collection only takes about half an hour.


Analysis
--------

If a database has any POSKeyErrors, we can find and repair those. Make sure the
ZEO server isn't running, as this will access the storage as a simple file
storage.

Installing those scripts is easy via a buildout section::

  [zodbdgc]
  recipe = zc.recipe.egg
  eggs = zc.zodbdgc

To check if any errors exist at all, we can run the following script::

  $ bin/multi-zodb-check-refs etc/zodb.conf

This can take about 15 to 30 minutes. Write down the reported errors, we'll
need them later on to analyze them.

The ``etc/zodb.conf`` might look like this::

  <zodb main>
    <filestorage 1>
      path <path to current dir>/var/filestorage/Data.fs
      blob-dir <path to current dir>/var/blobstorage
    </filestorage>
  </zodb>

If there are some errors, we can create a reference database to make it easier
to debug and find those errors::

  $ bin/multi-zodb-check-refs -r var/refdb.fs etc/zodb.conf

This is significantly slower and can take several hours to complete. Once this
is complete you can open the generated database. Get an interpreter with the
ZODB available in it. For example::

  [zeopy]
  recipe = zc.recipe.egg
  eggs =
      ZODB3
      zc.zodbdgc
  interpreter = zeopy
  scripts = zeopy

Then we can run the interpreter::

  $ bin/zeopy

  >>> import ZODB.config
  >>> db = ZODB.config.databaseFromFile(open('./etc/refdb.conf'))
  >>> conn = db.open()
  >>> refs = conn.root()['references']

The ``etc/refdb.conf`` might look like this::

  <zodb main>
    <filestorage 1>
      path <path to current dir>/var/refdb.fs
    </filestorage>
  </zodb>

If you've gotten this error report::

  !!! main 13184375 ?
  POSKeyError: 0xc92d77

You can look up the poid it was referenced from via::

  >>> parent = list(refs['main'][13184375])
  >>> parent
  [13178389]

If you prefer the hex representation, that's easy as well::

  >>> from ZODB.utils import p64
  >>> p64(parent[0])
  '\x00\x00\x00\x00\x00\xc9\x16\x15'

Once you are finished you should be nice and close the database::

  >>> conn.close()
  >>> db.close()

With this information, you should get back to your actual database and look
up this object. Leave the ref db open, as you might need to recursively look
up some more objects, until you get one you can identify and work on.

You could load the parent. In a debug prompt you could do something like::

  >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
  2010-04-28 14:28:28 ERROR ZODB.Connection Couldn't load state for 0xc91615
  Traceback (most recent call last):
  ...
  ZODB.POSException.POSKeyError: 0xc92d77

Gah, this gives us the POSKeyError of course. But we can load the actual data
of the parent, to get an idea of what this is::

  >>> app._p_jar.db()._storage.load('\x00\x00\x00\x00\x00\xc9\x16\x15', '')
  ('cBTrees.IOBTree
  IOBucket
  q\x01.((J$KT\x02ccopy_reg
  _reconstructor
  q\x02(cfive.intid.keyreference
  KeyReferenceToPersistent
  ...

Now we can be real evil and create a new fake object in place of the missing
one::

  >>> import transaction
  >>> transaction.begin()

The poid that was reported missing was ``13184375``::

  >>> from ZODB.utils import p64
  >>> p64(13184375)
  '\x00\x00\x00\x00\x00\xc9-w'

  >>> from persistent import Persistent
  >>> a = Persistent()
  >>> a._p_oid = '\x00\x00\x00\x00\x00\xc9-w'

We cannot use the ``add`` method of the connection, as this would assign the
object a new poid. So we replicate its internals here::

  >>> a._p_jar = app._p_jar
  >>> app._p_jar._register(a)
  >>> app._p_jar._added[a._p_oid] = a

  >>> transaction.commit()

Both getting the object as well as its parent will work now::

  >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9-w')
  <persistent.Persistent object at 0xa3e348c>

  >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
  BTrees.IOBTree.IOBucket([(39078692, <five.intid.keyreference...