AI summary by TwinMind
Deploying On Fridays
Theo - t3․gg
8m 1s
Friday Deployment Problem
- Friday deployments = common industry pain point (meme + real problem)
- Limited window to fix issues on "chill" day
- Risk of working into weekend if not fixed in time
- Growing frequency of major outages (GitHub, Cloudflare examples)
Types of Changes: Easy vs Hard Rollback
- Easy to rollback examples:
- Vercel deployments: one-click rollback, instant redeploy options
- PlanetScale: 30-minute rollback window, data preserved in both old/new tables
- Hard to rollback examples:
- npm packages: can't roll back published versions
- Must publish newer version to override (6.0.1 → 6.0.2)
- Bad major releases require rushing fixes or redeploying old versions as new
- "No scenario where bad npm release doesn't suck immensely"
UploadThing Deployment Strategy
- Frontend/infrastructure through Vercel
- Database changes through PlanetScale
- Won't do major package releases on Fridays due to rollback difficulty
- Different treatment for rollback-capable vs non-rollback systems
Safety Nets vs Guard Rails Philosophy
- Guard rails: unit tests, code review (prevent mistakes before deployment)
- Safety nets: rollback mechanisms, error reporting (recover after mistakes happen)
- Industry over-focuses on guard rails, under-invests in safety nets
- Guard rails don't help once mistake is live in production
Bug Statistics Reality
- Up to 70 bugs per 1000 lines of code
- 15+ bugs per 1000 lines reach users despite code review/tests
- "Doesn't matter how good tests are, you're going to ship bugs"
- Friday deployments = Friday outages inevitably
Essential Safety Net Components
- Easy/trivial rollback mechanisms
- Fast error reporting (Sentry, Axiom, Highlight)
- Build pipeline <5 minutes for quick fixes
- Seconds-level rollback capability when possible
On-Call Culture Recommendations
- All engineers should participate in on-call process
- Especially valuable for junior developers
- Learn what's fragile, where bugs happen, fix processes
- Shows how process works when code fails
- "No better way to see what isn't working at your company"
Production Outage Training
- Recommend engineers have production outage in first 1-2 weeks
- Gets it "out of your system"
- Reduces anxiety about causing problems
- Familiarizes with bug-fixing processes
- Problems will happen regardless - better to be prepared
Process Improvements
- Override slow code review for urgent fixes
- Good internal culture/documentation around outages
- Accept that bugs happen rather than panic
- Different treatment for different types of deployments based on rollback capability
