OVH SBG postmortem

Hello,
This morning at 7:23, we had a major breakdown on our site in Strasbourg (SBG): a power outage that put in the dark our 3 datacentres SBG1, SBG2 and SBG4 for 3:30. The worst scenario that can happen to us.

The SBG site is powered by a 20KVA power line consisting of 2 cables that each deliver 10MVA. The two cables work together, and are connected to the same source and on the same circuit breaker at ELD (Strasbourg Electricité Réseaux). This morning, one of the two cables was damaged and the circuit breaker shut down the data centers.

The SBG site is intended to operate, without time limit, on generators. For SBG1 and SBG4, we set up a first system of 2 generators of 2MVA each, configured in N + 1 and 20KV. For SBG2, we set up 3 N + 1 groups of 1.4MVA each. In the event of a break in the external source, the high voltage cells are automatically reconfigured by a motorized tilt system. In less than 30 seconds, the SBG1, SBG2 and SBG4 data centers are re-powered in 20KV. To allow all these scales without cutting the power of servers, we have inverters (UPS) knowing operate without power for 8 minutes.

This morning, the motorized tilt system did not work. The start order of the groups was not given by the PLC. It is an NSM (Normal Power Saving) machine supplied by the supplier of high-voltage 20KV cells. We are in contact with him, in order to understand the origin of this dysfunction. However, it is a defect that should have been detected during periodic fault simulation tests on the external source. SBG's last test of group testing is from the end of May 2017. During this last test, we fed SBG only from the gensets during 8H without any problem and each month we test the empty groups. And despite everything, the whole of this device was not enough today to avoid this breakdown.

Around 10am, we managed to switch the cells manually and we started feeding the datacentre again from the generators. We have asked ELD to disconnect the faulty cable from the high voltage cells and reset the circuit breaker with only 1 of the 2 cables, thus limiting it to 10MVA. The manipulation was carried out by ELD and the site was re-fed around 10:30. The SBG routers were reachable from 10:58.

Since then, we have been working on getting services back on track. Powering the site with energy makes it possible to restart the servers, but it remains to restart the services running on the servers. That is why each service comes back gradually since 10h58. Our monitoring system allows us to know the list of servers that have started successfully and those that still have a problem. We work on each of these servers to identify and solve the problem that prevents it from restarting.

At 7:50, we set up a crisis unit at RBX, where we centralized the information and actions of all the teams. A truck from RBX was loaded with spare parts for SBG. He arrived at his destination around 5.30 pm Our local teams have been reinforced by teams from the data center of LIM in Germany and RBX, they are all mobilized on site since 16H00. Currently, more than 50 technicians work at SBG to get all services back on track. We are preparing the work of that night and, if necessary, of tomorrow morning.

Take a step back. To avoid a disaster scenario of this type, during the last 18 years, OVH has developed electrical architectures capable of withstanding all kinds of electrical incidents. Every test, every little flaw, every new idea has enriched our experience, which allows us to build reliable datacentres today.

So why this failure? Why SBG did not resist a simple power outage of ELD? Why all the intelligence that we developed at OVH, did not prevent this failure?

The quick answer: SBG's power grid inherited design flaws related to the initially low ambition for the site.

The long answer:
In 2011, we planned the deployment of new data centers in Europe. To test the appetite of each market, with new cities and new countries, we imagined a new data center deployment technology, based on maritime containers. Thanks to this technology, developed in-house, we wanted to have the flexibility to deploy a data center without the time constraints associated with building permits. Originally, we wanted to be able to validate our hypotheses before investing sustainably in a site.