10am Saturday morning, and the following message arrives from our excellent data centre hosts Custodian Data Centres:
"ME14 has seen a complete loss of 11kV supplies across all feeds due to two separate off-site high-voltage cable faults in the local area of Maidstone.
"CDC's UPS systems carried the load while all on-site generation started; and then transferred on to load as expected without any interruption to critical infrastructure."
That's not a good morning for their sysadmin team. We suspect a lot of tea hit a monitor somewhere in Maidstone as that happened. There was a moment of tension on the Sandgate area, despite the reassurances!
However despite the failure of TWO power lines into the building, they had UPS systems that took over immediately and kept their Maidstone centre running: we didn't even notice a power flick, and services were 100% available.
Prater Raines host servers at both their Maidstone and Dartford data centres. We have failover between both of them so for most services were pretty much covered (as long as the power also stayed on at Dartford, which it did).
However, we were watching, as were they, until they could issue the "all-clear" for Maidstone an hour later:
"UKPN engineers have completed works to repair the supplies and we have seen the restoration of our 11kV feeders now. The site has cleanly transferred back from local generation to grid without interruption to critical load."
Nothing to see here. Our web services didn't even glitch, because of the redundancy Custodian build into their systems, on top of the redundancy we build into ours. Their comms were great throughout. And that's why we like hosting with them.