AI summary by TwinMind

How CloudFlare Crashed NPM

Theo - t3․gg

10m 59s

Outage Overview

  • Cloudflare outage October 30, 2023 lasting 37 minutes
  • Caused by misconfiguration of deployment tool used by Worker KV
  • Multiple core services affected: npm, GitLab, various websites
  • Worker KV returned HTTP 401 errors instead of requested key-value pairs

Worker KV Impact and Services Affected

  • Worker KV: globally distributed key-value store used for configuration data, routing lookups, static asset bundles, authentication tokens
  • Used by both customers and Cloudflare teams internally
  • Services impacted:
    • Pages: applications unreachable, returned HTTP 500 errors, new deployments failed
    • Access: unauthenticated users couldn't log in, JWT validation failed, device posture policies failed
    • Images API: returned errors
    • Most core Cloudflare features down except DDoS protection

Incident Timeline

  • 19:39: Worker KV team began progressive deployment of new build
  • 19:44: Internal progressive deployment API returned staging build to production
  • 19:54: New build rolled to percentage of traffic, routed to wrong destination, alerts triggered, rollback decision made
  • 20:01: Rollback attempted, traffic starts failing at scale
  • 20:15: Engineers manually edited via break glass mechanisms, reverted to last known good build
  • 20:31: Error rates return to normal, impact resolved
  • 21-minute delay between issue detection and successful rollback

Root Cause Analysis

  • Deployment tool bug: returned broader list of releases than intended
  • Production and staging releases returned together instead of single environment
  • Service deployed/tested in staging but deployment automation bug caused production to reference staging script
  • Result: 50% of traffic directed to non-existent version, effectively "black holing" traffic
  • Worker KV became unreachable, calls directed to unauthorized version returning 401 errors

Resolution Challenges

  • Internal dependency problem: Cloudflare Access (authentication layer) built on Worker KV
  • Couldn't access internal tools including dashboard needed for rollback
  • Had to use break glass mechanisms to bypass normal tooling
  • Insufficient testing of rollback mechanisms acknowledged

Systemic Issues Identified

  • Over-reliance on internal products: Worker KV underpins tremendous amount of Cloudflare products/services
  • Centralization risk: Cloudflare deeply "dogfoods" their technology (unlike Google Cloud)
  • Blast radius: Critical dependencies affect multiple services simultaneously
  • Deployment tooling: Current design assumes single tenant model, needs multitenancy support

Remediation Actions Planned

  • Onboard KV deployments to standardized worker deployment models with automated impact detection/recovery
  • Ensure rollback process has access to known good deployment identifier when Cloudflare Access is down
  • Add pre-checks to deployments validating input parameters to prevent version mismatches
  • Harden progressive deployment tooling for multitenancy design
  • Add validation to progressive deployment scripts verifying deployment matches app environment (production/staging)

Industry Impact Discussion

  • NPM down for 30+ minutes (rare occurrence)
  • 70%+ of internet covered by Cloudflare and AWS combined
  • Centralization vs reliability trade-offs
  • Balance needed between relying on services vs rolling own infrastructure

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome