Google Cloud outage shows it’s still hard to understand hyperscalers’ real resilience regimes

Summary

off prem Google Cloud outage shows it’s still hard to understand hyperscalers’ real resilience regimes Single datacenter and just three services taken down by ‘upstream’ power problem, while the rest of a zone and region kept humming Google Cloud last week experienced an outage that analysts say demonstrates that not all promises of cloudy resilience are created equal.Google’s incident report explained that three services – the VMware Engine (GCVE), NetApp Volumes, and Bare Metal Solutions (BMS) – experienced a 15-hour outage due to a cooling failure in its europe-west4-a zone.The report includes the following detail: “The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure.” The important detail there is that Google uses a discrete datacenter for those three services. Another notable element of the incident report is the admission that “An electrical fault occurred on the utility grid upstream of the datacenter, disrupting the electrical distribution gear and cooling equipment.”Google hasn’t explained how an upstream failure caused that disruption but did say it “proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment.”Whenever your correspondent talks to hyperscalers or datacenter operators about how they ensure resilience, they tell me about their use of multiple redundant pieces of energy infrastructure, plus on-site generation capabilities that can keep a datacenter powered for days if necessary. We’ve asked Google if it had generators or other energy sources at this site, and if so, why it nonetheless had to turn down workloads. We’ve not received a response at the time of writing. Google told us its incident analysis “is currently ongoing” and promised to follow up once it is available.Hidden dependenciesWe also asked Google if it advertises the fact that some of its services are tied...