Cloud & DevOpsNote
Google Cloud's 6-hour London outage began with generators that got no start signal
Google's final report of 4 November on europe-west2-b: how sensors above an open breaker kept the generators idle, and why 80 minutes without power meant six hours of disruption.
A Google Cloud data centre in London lost all power on 20 October because its generators were never told to start. One of the site’s two mains feeds was already out for repair when a cable fault took the other, and sensors wired upstream of the open breaker still read live voltage on the feed under repair. The controller concluded that mains power was healthy and sent no start command.1 Google’s final report, posted on 4 November, puts the disruption in its London region, europe-west2, at 6 hours 8 minutes and lists 40 affected services.1
6h 08m
disruption across the region, 07:00 to 13:08 London time
7 min
from mains failure to the batteries running out
Google timeline
80 min
with no power in the data halls
Google timeline
40
Google Cloud services on the report's impact list
01The controller that saw a healthy feed
The site has two independent utility feeds, A and B, a manual standby connection, and diesel generators run by a pair of programmable logic controllers (PLCs). Batteries give about five minutes, enough for a PLC to detect a failure, start the generators and move the load.1
A quarterly generator test on 14 October found issues that led to repairs on feeder B, so its breaker was opened and the whole site ran on feed A. At 06:53 London time an underground cable fault on the utility’s network took feed A out. The PLCs’ undervoltage inputs were wired on the line side of the open breaker, where feed B was still live. Google calls this “a voltage-sensing design flaw and PLC misconfiguration”: the logic never checked breaker position as well as voltage.1
The redundancy did not help, because the active and standby PLCs ran identical logic and read the same sensors. When staff tried a remote load transfer, the system blocked that too, on the same reading. They moved the site to the standby utility instead, but the batteries ran out first.1
02Why 80 minutes without power became six hours
Power was back in the halls after 80 minutes; bringing the zone back took four hours more
Google's timeline for zone europe-west2-b, 20 October 2026, in UTC (London was UTC+1)
Show tableHide table
| Date | Event | Detail |
|---|---|---|
| 20 Oct, 05:53 UTC | Feed A fails | No generator start command; batteries carry the site |
| 20 Oct, 06:00 UTC | Batteries run out | Hosts, storage and network in the zone lose power |
| 20 Oct, 06:54 UTC | Standby feed restored | Power restoration to the data hall begins |
| 20 Oct, 07:20 UTC | Halls powered | Utility power re-established to the data halls |
| 20 Oct, 07:50 UTC | Disks start recovering | Storage with unclean shutdowns begins recovery; responders hold back services |
| 20 Oct, 10:45 UTC | Traffic returns | Gradual, controlled re-admission to the zone |
| 20 Oct, 11:29 UTC | Core disruption ends | Regional services and compute management fully undrained |
| 20 Oct, 12:08 UTC | 99.5% of VMs healthy | End of the 6 hours 8 minutes |
Source: Google Cloud Service Health: incident report of 4 November 2026 · as of 10 November 2026 · Events close in time are spread apart so their labels stay legible
Power loss that abrupt left storage devices with unclean shutdowns. Responders “chose not to restore services that would otherwise make this worse”, and Google says the volume of devices meant recovery took “a substantial period” before customers could return.1
03What running in several zones did and did not buy
Google treats the loss of anything held entirely in the zone as “working within the design of a zonally-scoped system”. Spreading across zones worked for some: Bigtable clusters with high-availability routing failed over automatically, Dataflow moved new batch jobs away within about a minute, and Cloud SQL high-availability instances failed over unless Regional Persistent Disk failures held them back.1
It did not work for others. Traffic moving into the other two zones saturated their capacity, and Google says some Compute Engine failovers were “unsuccessful due to capacity constraints”. AlloyDB regional primaries waited up to 95 minutes to start failing over, and about 4% stalled. Apigee X returned errors for 7 hours 19 minutes because a scheduling misconfiguration had put two of three datastore replicas in the lost zone. About 45% of Looker instances stopped responding. Google names two causes beyond the zone: “demand surges” on neighbouring zones and “control-plane dependencies across zonal boundaries”. It found no impact outside europe-west2.1
Google gives no customer count; The Register named Lloyds, HSBC, Royal Mail and Revolut among those disrupted.2 The repair is still running. Google says the PLCs were reconfigured and tested by 26 October, but its date for the utility’s cable repair moved from 30 October in the report of 27 October to 13 November, and the site’s return to feeds A and B from 31 October to 17 November. It expects to finish checking PLC configurations at similar sites by 15 November.1
Sources
Disclosure. TechBag is a software reseller: it earns a margin when a reader buys through it, including from vendors named here. The analysis is TechBag’s own; no vendor paid for or saw this piece before publication. Facts are labelled reported, TechBag count or our view.
Keep reading
