AI summary by TwinMind

Why the internet went down for 2.5 hours yesterday

Theo - t3․gg

32m 5s

Outage Overview

  • Major internet services down simultaneously: Discord, Google Cloud, Spotify, Google, Twitch, Character AI, Google Meet, Rocket League, Nest, Cloudflare, AWS, T3 chat
  • Exposed concerning consolidation of internet infrastructure around same providers
  • Particularly scary due to interdependencies between major cloud providers

PostHog Sponsorship Segment

  • T3 chat uses PostHog for analytics, feature flags, surveys
  • Feature flag implementation for T3 chat beta toggle shown in code
  • 10,000 people opted into beta
  • 90%+ of PostHog users don't pay (free tier sufficient)

Cloud Provider Landscape

  • Four biggest web hosting providers: AWS, Google Cloud, Azure (Microsoft), Cloudflare
  • Azure unaffected by outage despite being second highest grossing cloud platform
  • AWS had some reported issues but unconfirmed
  • Twitch outage was self-caused, separate from cloud provider issues

Cloudflare Service Architecture

  • Three eras of Cloudflare:
    • Era 1: DDoS protection (classic DNS protection service)
    • Era 2: Developer platform (Workers, KV, R2, Turnstile)
    • Era 3: Zero trust (corporate security platform)
  • Developer platform services:
    • Worker KV: key-value store (rock solid, industry-relied upon)
    • Workers: JS runtime on edge (cheaper than Lambda)
    • CDN: content distribution (very tried and true)
    • R2: blob store S3 alternative (solid but has weird edges)
    • D1: database product (too limited, not recommended)
    • Turnstile: captcha replacement (unreliable in invisible mode)

Root Cause Analysis

  • Cloudflare Worker KV has dependency on Google Cloud for cold storage
  • This dependency was largely unknown to public and developers
  • Worker KV serves as core infrastructure for many Cloudflare services
  • When Worker KV fails, cascades to vast majority of Cloudflare services
  • Comparison to RAM being removed from computer - essential platform

Outage Impact Details

  • Duration: 2 hours 28 minutes
  • Globally impacted 100% of users on affected services
  • Worker KV: 90%+ of requests failing
  • Services affected:
    • Access: entirely down (identity-based system)
    • Gateway: certain queries affected
    • Warp: affected due to Access/Worker KV dependency
    • Dashboard: major issues, standard login failed, Google OAuth failed
    • Turnstile/Challenges: down (would fail bots AND humans)
    • Images: 100% upload failure rate at peak
    • Stream: entirely down
    • Workers AI: entirely down
    • Pages: peaked at 100% failure
    • Workers: largely down, especially uploads/changes
    • CDN: mostly up with random issues

Technical Architecture Issues

  • Worker KV designed as "coreless service" - no single point of failure goal
  • Currently relies on central data store for source of truth
  • Single source of truth makes orchestration easier but hurts reliability
  • Cold storage dependency on Google Cloud exposed during outage
  • Most data eventually needs cold storage retrieval in KV architecture

Cloudflare's Response Timeline

  • 17:52 UTC: Warp issues with new device spin-up
  • 18:05 UTC: Access team alerted (13 minutes later)
  • 18:06 UTC: Root cause identified as Worker KV, upgraded to P1
  • 18:21 UTC: Bumped to P0 (highest severity, all hands on deck)
  • 18:43 UTC: Access explores removing Worker KV dependency
  • 19:09 UTC: Gateway begins graceful degradation
  • Parallel efforts: reduce load, deploy alternative backing store
  • Recovery began when Google Cloud storage infrastructure recovered

Transparency and Ownership

  • Exceptional level of ownership from Cloudflare CTO Dane Knecht
  • Public apology acknowledging customer impact
  • Full postmortem published within 4 hours of resolution
  • Avoided blaming Google directly, took full responsibility
  • Shared actual error rate charts and detailed timeline
  • Contrast with Google's approach (doesn't dogfood their own products)

Remediation Plans

  • Accelerating existing work to remove singular dependency on external storage
  • Improving redundancy on Worker KV storage infrastructure
  • Backfilling internal infrastructure to own systems
  • Blast radius reduction for individual products
  • Tools for progressive namespace re-enabling during incidents
  • Goal: make each service resilient to single point of failure

Lessons Learned

  • Importance of understanding full dependency chain
  • Risk of building on platforms with hidden dependencies
  • Value of transparency and ownership during incidents
  • Cache invalidation can cause self-DDoS when cold storage overwhelmed
  • Parallel mitigation strategies crucial during major incidents

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome