Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Overview
- ========
- This is the storage server for the XXX website.
- Production
- ----------
- This buildout sets up a ZEO storage server. Make sure to run this buildout
- with the same Python version, as you run the ZEO clients.
- Currently this sets up a ZEO 3.9 server with one filestorage and a blobstorage.
- To run this buildout you do::
- $ python2.4 bootstrap.py
- $ bin/buildout
- This sets up the ZEO server, supervisord, log rotation and configures cron
- entries to restart and do regular maintenance. The ZEO server is protected via
- an authentication database.
- Development
- -----------
- If you want to test this locally and do some development, you can use the
- ``devel.cfg`` buildout file::
- $ python2.4 bootstrap.py
- $ bin/buildout -c devel.cfg
- This will avoid creating crontab entries or log rotation functionality.
- Garbage collection
- ------------------
- We don't do garbage collection as part of a pack anymore. Instead we use
- explicit garbage collection via a different job::
- bin/multi-zodb-gc etc/zeo.conf
- Garbage collection can take between one and two hours.
- Packing
- -------
- We do regular packing every day, via::
- bin/zeopack
- Packing without garbage collection only takes about half an hour.
- Analysis
- --------
- If a database has any POSKeyErrors, we can find and repair those. Make sure the
- ZEO server isn't running, as this will access the storage as a simple file
- storage.
- Installing those scripts is easy via a buildout section::
- [zodbdgc]
- recipe = zc.recipe.egg
- eggs = zc.zodbdgc
- To check if any errors exist at all, we can run the following script::
- $ bin/multi-zodb-check-refs etc/zodb.conf
- This can take about 15 to 30 minutes. Write down the reported errors, we'll
- need them later on to analyze them.
- The ``etc/zodb.conf`` might look like this::
- <zodb main>
- <filestorage 1>
- path <path to current dir>/var/filestorage/Data.fs
- blob-dir <path to current dir>/var/blobstorage
- </filestorage>
- </zodb>
- If there are some errors, we can create a reference database to make it easier
- to debug and find those errors::
- $ bin/multi-zodb-check-refs -r var/refdb.fs etc/zodb.conf
- This is significantly slower and can take several hours to complete. Once this
- is complete you can open the generated database. Get an interpreter with the
- ZODB available in it. For example::
- [zeopy]
- recipe = zc.recipe.egg
- eggs =
- ZODB3
- zc.zodbdgc
- interpreter = zeopy
- scripts = zeopy
- Then we can run the interpreter::
- $ bin/zeopy
- >>> import ZODB.config
- >>> db = ZODB.config.databaseFromFile(open('./etc/refdb.conf'))
- >>> conn = db.open()
- >>> refs = conn.root()['references']
- The ``etc/refdb.conf`` might look like this::
- <zodb main>
- <filestorage 1>
- path <path to current dir>/var/refdb.fs
- </filestorage>
- </zodb>
- If you've gotten this error report::
- !!! main 13184375 ?
- POSKeyError: 0xc92d77
- You can look up the poid it was referenced from via::
- >>> parent = list(refs['main'][13184375])
- >>> parent
- [13178389]
- If you prefer the hex representation, that's easy as well::
- >>> from ZODB.utils import p64
- >>> p64(parent[0])
- '\x00\x00\x00\x00\x00\xc9\x16\x15'
- Once you are finished you should be nice and close the database::
- >>> conn.close()
- >>> db.close()
- With this information, you should get back to your actual database and look
- up this object. Leave the ref db open, as you might need to recursively look
- up some more objects, until you get one you can identify and work on.
- You could load the parent. In a debug prompt you could do something like::
- >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
- 2010-04-28 14:28:28 ERROR ZODB.Connection Couldn't load state for 0xc91615
- Traceback (most recent call last):
- ...
- ZODB.POSException.POSKeyError: 0xc92d77
- Gah, this gives us the POSKeyError of course. But we can load the actual data
- of the parent, to get an idea of what this is::
- >>> app._p_jar.db()._storage.load('\x00\x00\x00\x00\x00\xc9\x16\x15', '')
- ('cBTrees.IOBTree
- IOBucket
- q\x01.((J$KT\x02ccopy_reg
- _reconstructor
- q\x02(cfive.intid.keyreference
- KeyReferenceToPersistent
- ...
- Now we can be real evil and create a new fake object in place of the missing
- one::
- >>> import transaction
- >>> transaction.begin()
- The poid that was reported missing was ``13184375``::
- >>> from ZODB.utils import p64
- >>> p64(13184375)
- '\x00\x00\x00\x00\x00\xc9-w'
- >>> from persistent import Persistent
- >>> a = Persistent()
- >>> a._p_oid = '\x00\x00\x00\x00\x00\xc9-w'
- We cannot use the ``add`` method of the connection, as this would assign the
- object a new poid. So we replicate its internals here::
- >>> a._p_jar = app._p_jar
- >>> app._p_jar._register(a)
- >>> app._p_jar._added[a._p_oid] = a
- >>> transaction.commit()
- Both getting the object as well as its parent will work now::
- >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9-w')
- <persistent.Persistent object at 0xa3e348c>
- >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
- BTrees.IOBTree.IOBucket([(39078692, <five.intid.keyreference...
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement