AI summary by TwinMind
Why the internet went down for 2.5 hours yesterday
Theo - t3․gg
32m 5s
Outage Overview
- Major internet services down simultaneously: Discord, Google Cloud, Spotify, Google, Twitch, Character AI, Google Meet, Rocket League, Nest, Cloudflare, AWS, T3 chat
- Exposed concerning consolidation of internet infrastructure around same providers
- Particularly scary due to interdependencies between major cloud providers
PostHog Sponsorship Segment
- T3 chat uses PostHog for analytics, feature flags, surveys
- Feature flag implementation for T3 chat beta toggle shown in code
- 10,000 people opted into beta
- 90%+ of PostHog users don't pay (free tier sufficient)
Cloud Provider Landscape
- Four biggest web hosting providers: AWS, Google Cloud, Azure (Microsoft), Cloudflare
- Azure unaffected by outage despite being second highest grossing cloud platform
- AWS had some reported issues but unconfirmed
- Twitch outage was self-caused, separate from cloud provider issues
Cloudflare Service Architecture
- Three eras of Cloudflare:
- Era 1: DDoS protection (classic DNS protection service)
- Era 2: Developer platform (Workers, KV, R2, Turnstile)
- Era 3: Zero trust (corporate security platform)
- Developer platform services:
- Worker KV: key-value store (rock solid, industry-relied upon)
- Workers: JS runtime on edge (cheaper than Lambda)
- CDN: content distribution (very tried and true)
- R2: blob store S3 alternative (solid but has weird edges)
- D1: database product (too limited, not recommended)
- Turnstile: captcha replacement (unreliable in invisible mode)
Root Cause Analysis
- Cloudflare Worker KV has dependency on Google Cloud for cold storage
- This dependency was largely unknown to public and developers
- Worker KV serves as core infrastructure for many Cloudflare services
- When Worker KV fails, cascades to vast majority of Cloudflare services
- Comparison to RAM being removed from computer - essential platform
Outage Impact Details
- Duration: 2 hours 28 minutes
- Globally impacted 100% of users on affected services
- Worker KV: 90%+ of requests failing
- Services affected:
- Access: entirely down (identity-based system)
- Gateway: certain queries affected
- Warp: affected due to Access/Worker KV dependency
- Dashboard: major issues, standard login failed, Google OAuth failed
- Turnstile/Challenges: down (would fail bots AND humans)
- Images: 100% upload failure rate at peak
- Stream: entirely down
- Workers AI: entirely down
- Pages: peaked at 100% failure
- Workers: largely down, especially uploads/changes
- CDN: mostly up with random issues
Technical Architecture Issues
- Worker KV designed as "coreless service" - no single point of failure goal
- Currently relies on central data store for source of truth
- Single source of truth makes orchestration easier but hurts reliability
- Cold storage dependency on Google Cloud exposed during outage
- Most data eventually needs cold storage retrieval in KV architecture
Cloudflare's Response Timeline
- 17:52 UTC: Warp issues with new device spin-up
- 18:05 UTC: Access team alerted (13 minutes later)
- 18:06 UTC: Root cause identified as Worker KV, upgraded to P1
- 18:21 UTC: Bumped to P0 (highest severity, all hands on deck)
- 18:43 UTC: Access explores removing Worker KV dependency
- 19:09 UTC: Gateway begins graceful degradation
- Parallel efforts: reduce load, deploy alternative backing store
- Recovery began when Google Cloud storage infrastructure recovered
Transparency and Ownership
- Exceptional level of ownership from Cloudflare CTO Dane Knecht
- Public apology acknowledging customer impact
- Full postmortem published within 4 hours of resolution
- Avoided blaming Google directly, took full responsibility
- Shared actual error rate charts and detailed timeline
- Contrast with Google's approach (doesn't dogfood their own products)
Remediation Plans
- Accelerating existing work to remove singular dependency on external storage
- Improving redundancy on Worker KV storage infrastructure
- Backfilling internal infrastructure to own systems
- Blast radius reduction for individual products
- Tools for progressive namespace re-enabling during incidents
- Goal: make each service resilient to single point of failure
Lessons Learned
- Importance of understanding full dependency chain
- Risk of building on platforms with hidden dependencies
- Value of transparency and ownership during incidents
- Cache invalidation can cause self-DDoS when cold storage overwhelmed
- Parallel mitigation strategies crucial during major incidents
