CloudFlare Killed Our Offsite Backups

This is the story of how our offsite backups suddenly vanished into the ether on summer morning, and we didn't notice for half a year.

We signed up for CloudFlare in March 2013 to use as a CDN. After a few tumultuous months of intermittent site outages when everything was working fine on our server and CloudFlare told us everything was fine on their end - and during which our only recourse was repeatedly shutting off CloudFlare to get our site running again - we threw the towel in in June and asked for a refund for all the downtime we'd experienced (and subsequent lost revenue). We're not huge, but we do in the low 6 figures a year in revenues. CloudFlare only agreed to refund the unused portion of our latest $200 / month business plan (guess we should've tried its free plan first), but at least it was something. I can't say we were happy, but we were just glad to be finished paying CloudFlare to increase our downtime, and soon found another CDN we liked better and that our site ran faster with, even if it didn't have the shiny-sounding DDoS protection CloudFlare offers (if it can keep your site running).

Anyway, that's not the interesting part - that's just the backstory.

At the time we signed up for CloudFlare, we'd been searching for a few months too for some kind of offsite backup service. CloudFlare conveniently had an upsell for a service named CodeGuard, which sounded perfect - for 5 bucks a month, CodeGuard would backup ALL of your files and MySQL databases, and allow you to restore pieces of these as needed from any point in time, beginning the moment you opened up your backups. Sounded like a dream.

We spent about a week going back and forth with CodeGuard and our host to get the settings just right that CodeGuard was whitelisted for database access to backup our DBs, but once it was set, we were able to trust in CodeGuard and forget backups entirely. CodeGuard regularly sent email updates letting us know our databases had been updated. In a worst case scenario - something or someone malicious made it onto our server and wreaked havoc - we were only a button push away from restoring everything just the way it was a day ago, via CodeGuard.

At some point late last year, I noticed these emails had stopped coming, when they'd been almost daily before, but just figured they were getting sorted into another of Gmail's new inboxes, and looked into it no further.

Several days ago, we suddenly had need to restore a few tables in our database to an earlier version. Seemed like the perfect job for CodeGuard! I'd been wanting to test drive the service. It was time to tap into our backups.

But when I tried logging into the CodeGuard site, I couldn't log in.

Did they change my password? I tried the forgot password option... only to be told there was no account for that email address. I didn't change the email on the account, did I?

I searched my inbox for "CodeGuard" - emails letting me know about our latest database updates suddenly just stopped on July 23rd.

No notification as to why. Nothing about our company card not charging (it's good until 2016, and never comes close to its limit), or about canceling our account.

They just ceased, as if CodeGuard had been swallowed up by a whale, or otherwise gone missing.

I wrote CodeGuard:

-------------------------

Hello,

Trying to access our CodeGuard account. Password wasn't working; now when I input my email address, I'm receiving a "no account found" error.

Doing a search for CodeGuard emails, it looks like we suddenly stopped receiving CodeGuard change updates in late July.

We never canceled, and I assumed we were still subscribed to CodeGuard and having our backups safely made. I'm very it was just a minor blip I was logging in to try and fix, and not a system meltdown. As far as I can tell, our account simply vanished... not something you want in a backup provider.

Can you shed any light on what's happened with our account?

Thank you

-------------------------

CodeGuard wrote back:

-------------------------

Hi,

Thanks for contacting CodeGuard support.

CodeGuard received a request from Cloudflare to remove your account when the CodeGuard app was turned off for your domain on June 15, 2013. If you did not turn it off intentionally, you will need to contact CloudFlare as they handle all billing, account setup, and deletion.

-------------------------

Well, that's interesting. We canceled and refunded from CloudFlare on June 3rd, and changed our nameservers to stop pointing to CloudFlare somewhere between the 3rd and the 6th of June.

And then on July 15th, CloudFlare asked CodeGuard to stop backing up our site (and, I suppose, trashcan our backups).

On July 23rd, CodeGuard went dark.

At no point did I cancel CodeGuard - only CloudFlare. I distinctly made sure we left our CodeGuard *open*.

At no point did I get a cancellation notice.

Our offserver backups simply vanished.

Fortunately, we turned out not to need to restore that table in the database anyway... so the story has a happy ending. And we make regular backups on the server itself, although that might not protect us in the event of a hacking. We're looking at Tarsnap right now.

But how scary is it to pay someone to handle your backups for you, think that's handled, and then suddenly those backups disappear?

What makes it worse is I was recommending CodeGuard left and right to fellow founders as essential for keeping one's site backed up. If you were one of those I recommended it to, my apologies for leading you astray.

The moral of the story seems to be this: no matter who's backing up your business offsite, make sure you've got a Plan C to that Plan B - because you have no control over that backup provider.

And like happened here with CodeGuard and CloudFlare, your backups very well may just vanish off into the summer night, with the toads and the lightning bugs.