Untitled

Received-SPF: pass (google.com: domain of 00000141ad522623-69cf65ab-1327-41ef-b9f8-c8899bb9101f-000000@amazonses.com designates 54.240.8.26 as permitted sender) client-ip=54.240.8.26;
User-Agent: Stallion
Date: Sat, 12 Oct 2013 15:39:04 +0000
From: "BuyVM" <[email protected]>
To: [email protected]
Reply-To: "[email protected]" <[email protected]>
Subject: =?utf-8?Q?[BuyVM_Las_Vegas]_Node23/38_Outages_October_9th-11th_(Wednesda?=
 =?utf-8?Q?y-Friday)?=
Message-ID: <00000141ad522623-69cf65ab-1327-41ef-b9f8-c8899bb9101f-000000@email.amazonses.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

On Wednesday, October 9th Node23 and 38 went offline due to a bad disks in
the array, this disk had approximately 4000 bad sectors on it and resulted
in at least 3 crashes before we stopped attempting to evacuate the node
while clients were online. We were able to get approximately 5 clients
migrated to another node before no further migrations were possible.

After repeated attempts to continue migrations both offline and online we
were forced to take the node completely offline and begin the 2~ days of
little sleep attempting to recover the bad drive. The first thing we did
even during migrations was replace one of the drives with a known good one
and allow the array to rebuild. This process only resulted in additional
crashes which forced us to try other means of recovery.

Having failed with the rebuild route both myself (Anthony) and Francisco
spent the next 6 hours taking shifts sleeping while ddrecover attempted to
recover whatever data it good from the problem disk. approximately halfway
through recovery the node began telling us the disk we were trying to
recover to was also bad. After replacing this new drive with 2 more disks
that were known to be good only to be told otherwise we had decided that the
RAID card itself was defective.

As a last option to prevent data loss we had contacted our datacenter and
requested that the entire array be moved to a new box so we could work from
a different RAID card, by the time we had paid the remote hand fees and got
the RAID back online it was too late as the client array would no longer
appear.

Both myself and Francisco are deeply sorry but despite our efforts we were
unable to recover any data, the only real good news to come out of this is
the fact that once we complete our upcoming SSD upgrades we will be on newer
generation RAID cards that we hope will be less susceptible to bad sectors.

Once again we are very sorry for this and also apologize for the extended
downtime as we were confident we would be able to recover the data. We have
already re-provisioned replacement containers which will require a root
password reset through http://manage.buyvm.net as our panel does not store
root passwords that can be used when reinstalling.

With that said, if there is anything further we can do to help turn this
into a better situation for you, please do not hesitate to open a ticket.

---
Anthony, Frantech Support
[IRC] irc.frantech.ca / #frantech
[Wiki] http://wiki.frantech.ca/