Actually I do see shared storage in real production in my day job...but it's all giant enterprise SAN replication, Veritas replicator, Oracle DB, and other impossibly expensive options.

I admit my goals here are completely pretentious.  I'm writing a job management system for my LEB empire to get rid of all the crontabs and allow some dependencies ("daily backup has completed on node X, validate and store on vault Y and if that fails, vault Z", "torrent has completed downloading here, move it there and initiate conversion and catalog, then notify this family member that her Linux ISO is ready", etc.)

I would like the master brain to be HA.  I'd really like it to be active-active so I don't have to worry about which one is active, failover, etc.

To do this I need the following:

- an HA database.  Strongly prefer SQL of some flavor since I'm an Oracle DBA by background.

- some kind of message queue.  Rabbit MQ is one option.  Since I'll have a daemon running on each node, I could really do anything - even SFTP back and forth.

- a shared filesystem, though I'm debating whether this is truly needed.  I might just go with something like Dropbox.  None of them work across a WAN very well and the only way to make them local is to put all your eggs in one provider's basket which makes me nervous.

I'm a little disappointed on the DB front.  With Oracle, MSSQL, etc. it's very easy to say "these two databases should be kept in sync, and here is the policy I'm choosing from an array of options - synchronous, eventual, etc."  Even across the WAN, async works pretty well.

For the DB, the options seem to be:

- Percona, which wants three for a quorum and doesn't like WANs.

- MySQL native multi-master replication, which seems to have all kinds of gotchas...typical MySQL crap:

http://scale-out-blog.blogspot.com/2012/04/if-you-must-deploy-multi-master.html

Yeah, I could remove auto increment and triggers but I thought we were a SQL database here...really, the "use a different auto increment" is one of the most ridiculous hacks I've ever heard of in my life.

- I haven't looked at Postgres

- I'm guessing there are non-SQL eventual consistency systems out there

- Maybe I should just use Google Cloud SQL or Amazon SimpleDB.