AI summary by TwinMind

Deploying On Fridays

Theo - t3․gg

8m 1s

Friday Deployment Problem

  • Friday deployments = common industry pain point (meme + real problem)
  • Limited window to fix issues on "chill" day
  • Risk of working into weekend if not fixed in time
  • Growing frequency of major outages (GitHub, Cloudflare examples)

Types of Changes: Easy vs Hard Rollback

  • Easy to rollback examples:
    • Vercel deployments: one-click rollback, instant redeploy options
    • PlanetScale: 30-minute rollback window, data preserved in both old/new tables
  • Hard to rollback examples:
    • npm packages: can't roll back published versions
    • Must publish newer version to override (6.0.1 → 6.0.2)
    • Bad major releases require rushing fixes or redeploying old versions as new
    • "No scenario where bad npm release doesn't suck immensely"

UploadThing Deployment Strategy

  • Frontend/infrastructure through Vercel
  • Database changes through PlanetScale
  • Won't do major package releases on Fridays due to rollback difficulty
  • Different treatment for rollback-capable vs non-rollback systems

Safety Nets vs Guard Rails Philosophy

  • Guard rails: unit tests, code review (prevent mistakes before deployment)
  • Safety nets: rollback mechanisms, error reporting (recover after mistakes happen)
  • Industry over-focuses on guard rails, under-invests in safety nets
  • Guard rails don't help once mistake is live in production

Bug Statistics Reality

  • Up to 70 bugs per 1000 lines of code
  • 15+ bugs per 1000 lines reach users despite code review/tests
  • "Doesn't matter how good tests are, you're going to ship bugs"
  • Friday deployments = Friday outages inevitably

Essential Safety Net Components

  • Easy/trivial rollback mechanisms
  • Fast error reporting (Sentry, Axiom, Highlight)
  • Build pipeline <5 minutes for quick fixes
  • Seconds-level rollback capability when possible

On-Call Culture Recommendations

  • All engineers should participate in on-call process
  • Especially valuable for junior developers
  • Learn what's fragile, where bugs happen, fix processes
  • Shows how process works when code fails
  • "No better way to see what isn't working at your company"

Production Outage Training

  • Recommend engineers have production outage in first 1-2 weeks
  • Gets it "out of your system"
  • Reduces anxiety about causing problems
  • Familiarizes with bug-fixing processes
  • Problems will happen regardless - better to be prepared

Process Improvements

  • Override slow code review for urgent fixes
  • Good internal culture/documentation around outages
  • Accept that bugs happen rather than panic
  • Different treatment for different types of deployments based on rollback capability

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome