When a large cloud provider has a bad day, the effects rarely stay inside the provider. Apps, websites and even smart devices that never mention the provider’s name stop working together. The best evidence for why comes from the providers’ own post-incident reports, and two of them are unusually detailed.
How clouds are meant to survive failure
Cloud platforms are built on the assumption that hardware fails. AWS divides its infrastructure into regions, and each region contains several Availability Zones, which it describes as one or more discrete data centers with separate and redundant power, networking and connectivity. The zones in a region are meaningfully distant from each other, AWS says up to about 60 miles, so that a flood or power failure in one does not take down the others, but close enough for fast replication. A well-built application spreads across zones, so losing one does not stop the service.
That design protects against physical failures very well. The incidents that make headlines tend to be a different kind: faults in software and configuration that every zone, or even every region, shares.
Case one: a DNS fault inside AWS
In October 2025, a disruption in AWS’s US East region began on the evening of 19 October and lasted until the afternoon of 20 October, Pacific time, with three distinct periods of customer impact. AWS’s post-event summary traces it to a latent race condition in the system that manages DNS records for its DynamoDB database service, which left the service’s regional address with an empty record. As we explain in our guide to how DNS works, once names stop resolving, healthy servers become unreachable.
The first fault was fixed in a few hours, yet the recovery was uneven. AWS says its server-launching system for EC2 depended on DynamoDB, and when DynamoDB returned, it entered what AWS calls congestive collapse and needed throttling and manual restarts. A backlog in network configuration then delayed connectivity for new instances, and health checks on load balancers began failing and removing capacity. Services including Lambda, container services and Amazon Connect saw errors or delays, and some Redshift clusters stayed impaired until the next day.
Case two: one bad record, everywhere
Google Cloud’s incident on 12 June 2025 has a different cause with a similar shape. Google’s incident report attributes it to Service Control, the component that checks quota and policy on API requests. A feature added in May contained a code path that was not protected by a feature flag and lacked proper error handling. It never ran in testing because it required a specific kind of policy data to trigger.
On 12 June, a policy change with unintended blank fields was inserted into a regional database and replicated globally within seconds. Service Control read the blank fields, hit an unguarded null pointer and crashed in a loop in every region. Google says engineers identified the cause within about ten minutes and had a kill switch rolling out within roughly forty. Restarting tasks then overloaded the underlying database because the code lacked randomized exponential backoff, which stretched recovery in its us-central1 region to about two hours and forty minutes. Google’s remediation list includes making the component fail open, requiring feature flags on critical binaries and auditing systems that consume globally replicated data.
What the two incidents have in common
Neither was an act of nature or an attack. Each began with a change or latent bug in software that many customers depend on. Each spread through a dependency: DNS for DynamoDB and then the services built on it, a globally replicated policy record for Google. And each was prolonged by recovery itself, with retries and restarts overwhelming systems that were trying to come back.
That is the general pattern. Shared services turn a local fault into a wide one, hidden layers hide the dependency from customers, and automatic retries can make a struggling system worse.
What customers can do
No design removes the risk, but several practices reduce it.
- Map your dependencies, including the third-party tools that themselves run on a particular cloud.
- Spread across zones as a baseline, and consider multiple regions for the most critical functions. AWS presents zones as protection against localized failures, which is why a fault shared across a whole region calls for more than multi-zone design.
- Plan for degraded service, deciding in advance which features can switch off while the core keeps running.
- Rehearse recovery, including the case where your own systems retry all at once.
- Communicate, with a status page and a plan for telling customers what is happening.
Multi-region and multi-provider designs cost more and add complexity of their own, so the sensible level of protection depends on what an hour of downtime costs.
For everyone else
Individuals can do little about a provider failure, but some habits help: keep essential documents available offline, know a second way to reach services such as banking, and check official status pages before assuming your own device is at fault.
