CHYP.ai · ORBIT // Cloudy_Day Security Suite

OUTAGE DISSECTOR · CASE FILE · Meltdown Index · vendor post-incident report

IAM configuration change propagated globally past canary

Google Cloud resolved M8.5 Meltdown 2020-04-08

Google Cloud: IAM configuration change propagated globally past canary. Scope: Global.

  • 50mDurationstart → resolved
  • 4Systems namedfrom the source
  • 7Places named7 on the map
  • 75Dependentscatalogued, inferred
  • 2/4Evidence kindsindependent families
  • 2Updatesofficial revisions

1Scrub the outage

full window
Severityofficial state Apr 8, 6 AMApr 8, 6 AMApr 8, 7 AMApr 8, 7 AMApr 8, 7 AMApr 8, 8 AM Official statusApr 8, 6:52 AM PT · Impact begins (vendor-stated)Apr 8, 7:42 AM PT · Mitigated (vendor-stated)Network & routingnothing in this windowEarly signalsnothing in this windowNews & chatternothing in this windowCoincidence (KEV)nothing in this window ▼ startresolved ▼

Drag or press Play to replay what was public at each moment. Each dot links to its evidence. Lanes are kept separate on purpose.

What happened

Google Cloud: IAM configuration change propagated globally past canary. Scope: Global.

What broke

Cloud IAM, Cloud SQL, BigQuery streaming, dependent Cloud APIs.

Who was hit

75 catalogued services run on Google Cloud (inferred). Hit: Global.

How it recovered

Resolved after 50m.

2Blast radius

Elastic Cloud · Degraded / minor incidentJFrog · MaintenanceAiven · OperationalAnthropic (Claude) · OperationalBigCommerce · OperationalBox · OperationalCalendly · OperationalCircleCI · OperationalClerk · OperationalCloudinary · OperationalCockroachDB Cloud · OperationalCodecov · OperationalCohere · OperationalDiscord · OperationalElevenLabs · OperationalExpo · OperationalFirebase · OperationalGmail · OperationalGoogle Drive · OperationalGoogle Meet · OperationalGoogle Workspace · OperationalGrafana Cloud · OperationalHubSpot · Operationalimgix · Operationalincident.io · OperationalJumpCloud · OperationalKinsta · OperationalLinear · OperationalMixpanel · OperationalMongoDB Atlas · OperationalMux · OperationalNetlify · OperationalPinecone · OperationalPlanetScale · OperationalRender · OperationalReplit · OperationalGoogle Cloud75 dependents +39 more catalogued dependents not drawn

Center: Google Cloud. Inner ring: what it runs on. Outer ring: catalogued services that run on it, coloured by their official status right now. Dependencies are inferred from public information.

3Blast map

North America · vendor-stated scope: globalSouth America · vendor-stated scope: globalEurope · vendor-stated scope: globalAfrica · vendor-stated scope: globalMiddle East · vendor-stated scope: globalAsia · vendor-stated scope: globalOceania · vendor-stated scope: global
  1. North America
  2. South America
  3. Europe
  4. Africa
  5. Middle East
  6. Asia
  7. Oceania

Places come from names in the public text (cities, PoP codes, countries, regions). Approximate.

4Failure-mode fingerprint

RetriesControl planeConfig pushCascadeCapacity
  • Config propagationvendor post-incident report (Meltdown Index)
  • Control-plane couplingvendor post-incident report (Meltdown Index)
  • Retry amplificationno sign
  • Dependency cascadeno sign
  • Capacity exhaustionno sign

Failure modes as stated in the vendor's own post-incident report (Meltdown Index).

5Evidence matrix

Official statusOfficial status · Apr 8, 12:52 AM PT: 0 itemsOfficial status · Apr 8, 1:09 AM PT: 0 itemsOfficial status · Apr 8, 1:26 AM PT: 0 itemsOfficial status · Apr 8, 1:43 AM PT: 0 itemsOfficial status · Apr 8, 2:00 AM PT: 0 itemsOfficial status · Apr 8, 2:17 AM PT: 0 itemsOfficial status · Apr 8, 2:34 AM PT: 0 itemsOfficial status · Apr 8, 2:51 AM PT: 0 itemsOfficial status · Apr 8, 3:08 AM PT: 0 itemsOfficial status · Apr 8, 3:25 AM PT: 0 itemsOfficial status · Apr 8, 3:42 AM PT: 0 itemsOfficial status · Apr 8, 3:59 AM PT: 0 itemsOfficial status · Apr 8, 4:17 AM PT: 0 itemsOfficial status · Apr 8, 4:34 AM PT: 0 itemsOfficial status · Apr 8, 4:51 AM PT: 0 itemsOfficial status · Apr 8, 5:08 AM PT: 0 itemsOfficial status · Apr 8, 5:25 AM PT: 0 itemsOfficial status · Apr 8, 5:42 AM PT: 0 itemsOfficial status · Apr 8, 5:59 AM PT: 0 itemsOfficial status · Apr 8, 6:16 AM PT: 0 itemsOfficial status · Apr 8, 6:33 AM PT: 0 itemsOfficial status · Apr 8, 6:50 AM PT: 1 itemOfficial status · Apr 8, 7:07 AM PT: 0 itemsOfficial status · Apr 8, 7:24 AM PT: 0 itemsNetwork & routingNetwork & routing · Apr 8, 12:52 AM PT: 0 itemsNetwork & routing · Apr 8, 1:09 AM PT: 0 itemsNetwork & routing · Apr 8, 1:26 AM PT: 0 itemsNetwork & routing · Apr 8, 1:43 AM PT: 0 itemsNetwork & routing · Apr 8, 2:00 AM PT: 0 itemsNetwork & routing · Apr 8, 2:17 AM PT: 0 itemsNetwork & routing · Apr 8, 2:34 AM PT: 0 itemsNetwork & routing · Apr 8, 2:51 AM PT: 0 itemsNetwork & routing · Apr 8, 3:08 AM PT: 0 itemsNetwork & routing · Apr 8, 3:25 AM PT: 0 itemsNetwork & routing · Apr 8, 3:42 AM PT: 0 itemsNetwork & routing · Apr 8, 3:59 AM PT: 0 itemsNetwork & routing · Apr 8, 4:17 AM PT: 0 itemsNetwork & routing · Apr 8, 4:34 AM PT: 0 itemsNetwork & routing · Apr 8, 4:51 AM PT: 0 itemsNetwork & routing · Apr 8, 5:08 AM PT: 0 itemsNetwork & routing · Apr 8, 5:25 AM PT: 0 itemsNetwork & routing · Apr 8, 5:42 AM PT: 0 itemsNetwork & routing · Apr 8, 5:59 AM PT: 0 itemsNetwork & routing · Apr 8, 6:16 AM PT: 0 itemsNetwork & routing · Apr 8, 6:33 AM PT: 0 itemsNetwork & routing · Apr 8, 6:50 AM PT: 0 itemsNetwork & routing · Apr 8, 7:07 AM PT: 0 itemsNetwork & routing · Apr 8, 7:24 AM PT: 0 itemsEarly signalsEarly signals · Apr 8, 12:52 AM PT: 0 itemsEarly signals · Apr 8, 1:09 AM PT: 0 itemsEarly signals · Apr 8, 1:26 AM PT: 0 itemsEarly signals · Apr 8, 1:43 AM PT: 0 itemsEarly signals · Apr 8, 2:00 AM PT: 0 itemsEarly signals · Apr 8, 2:17 AM PT: 0 itemsEarly signals · Apr 8, 2:34 AM PT: 0 itemsEarly signals · Apr 8, 2:51 AM PT: 0 itemsEarly signals · Apr 8, 3:08 AM PT: 0 itemsEarly signals · Apr 8, 3:25 AM PT: 0 itemsEarly signals · Apr 8, 3:42 AM PT: 0 itemsEarly signals · Apr 8, 3:59 AM PT: 0 itemsEarly signals · Apr 8, 4:17 AM PT: 0 itemsEarly signals · Apr 8, 4:34 AM PT: 0 itemsEarly signals · Apr 8, 4:51 AM PT: 0 itemsEarly signals · Apr 8, 5:08 AM PT: 0 itemsEarly signals · Apr 8, 5:25 AM PT: 0 itemsEarly signals · Apr 8, 5:42 AM PT: 0 itemsEarly signals · Apr 8, 5:59 AM PT: 0 itemsEarly signals · Apr 8, 6:16 AM PT: 0 itemsEarly signals · Apr 8, 6:33 AM PT: 0 itemsEarly signals · Apr 8, 6:50 AM PT: 0 itemsEarly signals · Apr 8, 7:07 AM PT: 0 itemsEarly signals · Apr 8, 7:24 AM PT: 0 itemsNews & chatterNews & chatter · Apr 8, 12:52 AM PT: 0 itemsNews & chatter · Apr 8, 1:09 AM PT: 0 itemsNews & chatter · Apr 8, 1:26 AM PT: 0 itemsNews & chatter · Apr 8, 1:43 AM PT: 0 itemsNews & chatter · Apr 8, 2:00 AM PT: 0 itemsNews & chatter · Apr 8, 2:17 AM PT: 0 itemsNews & chatter · Apr 8, 2:34 AM PT: 0 itemsNews & chatter · Apr 8, 2:51 AM PT: 0 itemsNews & chatter · Apr 8, 3:08 AM PT: 0 itemsNews & chatter · Apr 8, 3:25 AM PT: 0 itemsNews & chatter · Apr 8, 3:42 AM PT: 0 itemsNews & chatter · Apr 8, 3:59 AM PT: 0 itemsNews & chatter · Apr 8, 4:17 AM PT: 0 itemsNews & chatter · Apr 8, 4:34 AM PT: 0 itemsNews & chatter · Apr 8, 4:51 AM PT: 0 itemsNews & chatter · Apr 8, 5:08 AM PT: 0 itemsNews & chatter · Apr 8, 5:25 AM PT: 0 itemsNews & chatter · Apr 8, 5:42 AM PT: 0 itemsNews & chatter · Apr 8, 5:59 AM PT: 0 itemsNews & chatter · Apr 8, 6:16 AM PT: 0 itemsNews & chatter · Apr 8, 6:33 AM PT: 0 itemsNews & chatter · Apr 8, 6:50 AM PT: 0 itemsNews & chatter · Apr 8, 7:07 AM PT: 0 itemsNews & chatter · Apr 8, 7:24 AM PT: 0 itemsCoincidence (KEV)Coincidence (KEV) · Apr 8, 12:52 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 1:09 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 1:26 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 1:43 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 2:00 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 2:17 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 2:34 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 2:51 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 3:08 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 3:25 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 3:42 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 3:59 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 4:17 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 4:34 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 4:51 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 5:08 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 5:25 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 5:42 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 5:59 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 6:16 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 6:33 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 6:50 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 7:07 AM PT: 0 itemsCoincidence (KEV) · Apr 8, 7:24 AM PT: 0 itemsApr 8, 12:52 AM PTApr 8, 7:42 AM PT

Correlation is not causation

Five independent lanes across the window. Brighter = stronger item. Items landing in the same column are coincidence unless the provider says otherwise.

6In the vendor's words

A latent issue with our canarying system allowed this specific type of configuration change to propagate globally in a rapid manner. As the configuration was pushed to production, the global Cloud IAM service quickly began to experience internal errors.

Engineers were alerted at 06:52 US/Pacific; the change was identified at 07:27 and the mitigation finished propagating by 07:42 for a majority of internal services. Some services (external Cloud IAM API, high-availability Cloud SQL, BigQuery streaming) took longer. The later end is not used as a single official close time here.

Verbatim excerpt from Google Cloud Service Health — IAM / Zanzibar incident, 8 Apr 2020.

7Historical twins

Nearest matches in the Control-Plane Meltdown Index (1.0.0, vendor-sourced, tracked since 2024 with earlier landmark cases). Similarity = provider + failure modes + scope.

Cloudy_Day take

  • A change went wider, faster than its safety net. Staged rollouts and kill switches exist for exactly this.
  • 75 catalogued services lean on one provider. That's concentration risk, and it's the real headline.
  • Longer than 4 of 19 indexed control-plane meltdowns with a stated duration.

Automated take: rules written by Cloudy_Day, applied to public data. Not a human verdict on this incident, not a root-cause finding, not affiliated with Google Cloud.

8Dissection capsule

Share card for this dissection

Everything above as one shareable, citable capsule with a content hash.

JSON capsuleMarkdownShare card (SVG)

Open Google Cloud in the War Room →Status page →

Embed this card
<a href="https://www.chyp.ai/dissect/m/gcp-iam-zanzibar-2020-04-08/"><img src="https://www.chyp.ai/dissect/m/gcp-iam-zanzibar-2020-04-08/card.svg" alt="IAM configuration change propagated globally past canary · ORBIT Outage Dissector" width="600"></a>

Sources & limits

  • Taken from the vendor's own post-incident report; quotes are verbatim excerpts.
  • Dependents are today's inferred catalogue, not who was affected on the day.
  • Correlation is not causation. Lanes are kept apart so overlap never reads as cause.
  • ORBIT is passive: it never contacts the provider or probes its systems. Independent researcher, not affiliated.