AI summary by TwinMind
How CloudFlare Crashed NPM
Theo - t3․gg
10m 59s
Outage Overview
- Cloudflare outage October 30, 2023 lasting 37 minutes
- Caused by misconfiguration of deployment tool used by Worker KV
- Multiple core services affected: npm, GitLab, various websites
- Worker KV returned HTTP 401 errors instead of requested key-value pairs
Worker KV Impact and Services Affected
- Worker KV: globally distributed key-value store used for configuration data, routing lookups, static asset bundles, authentication tokens
- Used by both customers and Cloudflare teams internally
- Services impacted:
- Pages: applications unreachable, returned HTTP 500 errors, new deployments failed
- Access: unauthenticated users couldn't log in, JWT validation failed, device posture policies failed
- Images API: returned errors
- Most core Cloudflare features down except DDoS protection
Incident Timeline
- 19:39: Worker KV team began progressive deployment of new build
- 19:44: Internal progressive deployment API returned staging build to production
- 19:54: New build rolled to percentage of traffic, routed to wrong destination, alerts triggered, rollback decision made
- 20:01: Rollback attempted, traffic starts failing at scale
- 20:15: Engineers manually edited via break glass mechanisms, reverted to last known good build
- 20:31: Error rates return to normal, impact resolved
- 21-minute delay between issue detection and successful rollback
Root Cause Analysis
- Deployment tool bug: returned broader list of releases than intended
- Production and staging releases returned together instead of single environment
- Service deployed/tested in staging but deployment automation bug caused production to reference staging script
- Result: 50% of traffic directed to non-existent version, effectively "black holing" traffic
- Worker KV became unreachable, calls directed to unauthorized version returning 401 errors
Resolution Challenges
- Internal dependency problem: Cloudflare Access (authentication layer) built on Worker KV
- Couldn't access internal tools including dashboard needed for rollback
- Had to use break glass mechanisms to bypass normal tooling
- Insufficient testing of rollback mechanisms acknowledged
Systemic Issues Identified
- Over-reliance on internal products: Worker KV underpins tremendous amount of Cloudflare products/services
- Centralization risk: Cloudflare deeply "dogfoods" their technology (unlike Google Cloud)
- Blast radius: Critical dependencies affect multiple services simultaneously
- Deployment tooling: Current design assumes single tenant model, needs multitenancy support
Remediation Actions Planned
- Onboard KV deployments to standardized worker deployment models with automated impact detection/recovery
- Ensure rollback process has access to known good deployment identifier when Cloudflare Access is down
- Add pre-checks to deployments validating input parameters to prevent version mismatches
- Harden progressive deployment tooling for multitenancy design
- Add validation to progressive deployment scripts verifying deployment matches app environment (production/staging)
Industry Impact Discussion
- NPM down for 30+ minutes (rare occurrence)
- 70%+ of internet covered by Cloudflare and AWS combined
- Centralization vs reliability trade-offs
- Balance needed between relying on services vs rolling own infrastructure
