Google Cloud outage triggered by engineer unplugging fibers

A routine maintenance task at Google Cloud’s us-central1-b region escalated into a multi-hour outage after an engineer inadvertently disconnected every fiber-optic cable in the affected zone. The incident, which lasted from 07:41 to 11:52 PT on September 1, left virtual machines unreachable and caused complete traffic flow disruption for resources in the impacted area.

What happened

During what was intended to be a standard hardware maintenance procedure, an engineer sequentially unplugged 100% of the fiber paths connecting network devices within 13 minutes. Google’s service health report noted that the disconnection bypassed built-in redundancy safeguards, which are designed to withstand single or even multiple device failures without affecting customer traffic. The company’s infrastructure relies on physical separation of devices and fiber paths, along with diverse power sources, to prevent such disruptions. However, the rapid and sequential disconnection of all cables prevented automated warnings from reaching the engineer before the damage was done.

Once the cables were unplugged, virtual machines in one zone of us-central1-b lost external connectivity entirely, though they retained internal communication. Google’s response involved rerouting traffic to healthy capacity elsewhere in the region, followed by physically reseating the disconnected fibers. Normal traffic flow resumed once the links were restored, but the outage lasted over four hours, with peak disruption reaching 100% packet loss for affected resources.

Why redundancy failed

Google’s cloud infrastructure is engineered to handle multiple simultaneous failures without customer impact. The system’s resilience is based on physical separation of routing devices and fiber paths, ensuring that no single point of failure can take down an entire zone. However, the incident revealed a critical vulnerability: human error during maintenance can override these safeguards if procedures are not followed precisely. In this case, the engineer’s actions sequentially disabled all redundant paths, effectively isolating the zone from the broader network.

The company’s preliminary root cause analysis emphasized that the disconnection was not a technical failure but a procedural one. Google’s maintenance protocols are designed to prevent such outcomes, but the engineer’s deviation from these protocols—whether due to oversight or miscommunication—resulted in the outage. The speed of the disconnection, completed within 13 minutes, further limited the system’s ability to trigger corrective alerts in time.

What Google is doing

Google has not disclosed specific corrective actions but indicated that the incident would prompt a review of maintenance procedures and training. The company’s service health report acknowledged that while the infrastructure itself performed as designed once the issue was identified, the procedural failure highlighted a need for additional safeguards. Technicians were able to restore service by physically reseating the fibers and redirecting traffic, but the incident underscored the risks of human error in highly redundant systems.

Broader implications

The outage serves as a reminder that even the most robust cloud infrastructure can be compromised by simple human mistakes. While Google’s redundancy measures are effective against hardware failures, they are not foolproof against procedural errors. The incident may prompt other hyperscale providers to reassess their own maintenance workflows, particularly in regions where physical redundancy is a cornerstone of reliability.

For customers, the event reinforces the importance of multi-region deployments for critical workloads. Even a single zone outage can have significant operational impacts, and relying on a single region—no matter how resilient—introduces avoidable risk.