Ultimate backup solution!

HOW TO MAKE FLAWLESS BACKUPS IN ABOUT 1 HOUR (incl. 45 minute coffee break)
---------------------------------------------------------------
1) Create your backup directory.  Mine is /backup and is a LVM LV that sits on top of amazon AWS.  (Easily and cheaply expandable.  I expand it as needed to keep AWS costs at a bare minimum (no over allocation)

2) Place this file there: http://pastebin.com/kxuiZ5M8  .. I called mine .validate_rsync

3) Create a unix user for each account, like "backup_SERVERNAME" with no password (not an empty password, just unable to complete a password auth.  You should be using priv key auth anyway.) .. You should also be setting nologin as the shell .. For the home directory, set to /backups/SERVERNAME/ (or, whatever) .. Create that directory with rx only permissons for owner only.

4) Create public/private keys for each server you want to backup on the server.  Store the private key in /root/.daily_backup_key and set appropriate permissions (!) ... Then, when you drop the authorized_keys file into each home directory on your backup server, which should be e.g. /backups/SERVERNAME/.ssh/authorized_keys .. make it look like this:

from="YOUR_BACKED_UP_SERVER_IP",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/backups/.validate_rsync" ssh-rsa <....your private key...>

See? All secure.  Each backed-up-server cannot see/access the other server's backups.  Each backed-up-server can only run rsync.

Now, what about the actual backups?  Well, this is the crappier part, and only for one reason.  I use getfacl (see man getfacl) to store permissions, because no backup solution in the world, it seemed, actually preserved permissions in a way that I could restore.  I tested restores with other solutions and, it didn't work. (Not for backing up multiple servers without letting each server able to access the next's backups -- e.g. isolation) So, I tried stuff until I found one that worked.  The unfortunate part is it has to traverse the whole directory tree getting permissions on each backup.  Oh, and getfacl/setfacl is loosernix only.  I use NetBSD and FreeBSD (and, just shut down my only OpenBSD) on a lot of my servers, so it presents a problem if you're in that situation.  If you're a linux-only guy, (1) Try another OS just for perspective, at least (2) You're in luck with my backup solution

5) So, here it is, just put this in e.g. /root/.backup_server and chmod +x:
http://pastebin.com/EvYXL8UM
Set it up a crontab.  NOTE: I usually run SSHD on port 4.  Please note you'll need to adjust the '-p ' parameter in the above file if you run it on the standard port or another non-standard port.  I hope enough people run it on a non-standard port to warrant keeping that in.

6) Real life stuff:

a) My backup includes mySQL specific handling so tables get locked for rsync'ing.  If you have very large, heavy-write (or heavy-change) tables that need to be 100% available, this may not be for you
b) Likewise, if you use pgSQL, heavy-write something-something... Please consider stopping/pausing/whatever those services while the server backs up.
c) getfacl will produce a lot of warnings for stuff like /proc.  and getfacl is NOT the ideal solution.  write a script to save permissions separately if you want to make it better
d) To restore a server, you'll need to create a new file system and copy the data over, and run setfacl using the permissions file it generated.  You'll also need to mkdev (or whatever the linux/your-distro equiv is.. I know mkdev is inside NetBSD, can't remember the linux equiv if it's not the same)
e) The absolute main thing is to test recovery of a server.  After your first complete backup, start a new virtual machine/server and try to restore that server from backups.  If it doesn't work, the backup solution is failed, and tweak/modify this one until it works, or find a new one.  A backup solution is only good as how well/easily it is restored.
f) There is always room for improvement.  This is a quick and dirty backup solution I whipped up to be as cheap (low disk space), fast (to write/implement), and easy to recover (Keep in mind, I use VPSs that I have root access to the host machine...so TEST YOUR RECOVERY)

It's not perfect but it works for me to backup dozens of VPSs powered by Xen and a few on-hardware servers.  When I shut down a server permanently, I tar -c | xz -9 -e the thing into a .tar.xz and download it onto a RAID1'd 2tb array I use for long term backups.

YMMV with any of it.  Take precautions.

I am a full time father/salesman/sysadmin/web developer/windows desktop app developer/networking consultant/pc repair tech/anything you can dream up.


My mail is B E N %

P.S. it took longer for me to write this than to develop the whole solution.

at % "p" "L" "z" "e" dot NET.