AI summary by TwinMind

Surprise $104,500 Bill? Breaking Down A BIG Mistake

Theo - t3․gg

27m 11s

Initial Billing Incident

  • Developer received $104,500 overdue bill from Netlify for simple static site
  • Initially assumed phishing scheme but was legitimate
  • Site had insane bandwidth usage totaling $114,000
  • Peak day: 60.7 terabytes traffic in single day
  • Site normally stayed within free tier, never exceeded 10GB monthly
  • Only 200 daily visitors typically
  • Attack focused on 3.4MB sound file hosted on site

Netlify's Initial Support Response Failure

  • Default offer: charge 20% of bill
  • "Generous" offer: only charge 5% = $5,000
  • Speaker calls this "absurd" and "incredibly unacceptable"
  • $5,000 bill for hobby project due to external abuse should not happen
  • Support response was biggest failure by company

Platform Protection Analysis

  • Platforms do have DDoS protection, not completely defenseless
  • Similar incident happened to speaker ~1.5-2 years ago on Vercel
  • Attackers targeted largest file (half megabyte JS bundle)
  • Vercel used opportunity to improve servers and filters
  • Speaker didn't pay anything due to relationship + public attention

Notification System Failures

  • Developer should have been alerted when spending skyrocketed
  • Claims no notifications received, only "extra usage package purchased" email
  • Speaker gets daily Vercel notifications for limit breaches
  • Balance needed: too many notifications train users to ignore them
  • 20%+ billing increases should trigger automatic notifications
  • Lack of notifications alone justifies bill forgiveness

File Hosting Best Practices

  • Large files shouldn't be hosted on Vercel/Netlify due to bandwidth costs
  • These platforms optimize for performance via global CDN
  • 3.4MB MP3 file inappropriate for these services
  • Alternatives: SoundCloud, GitHub assets, upload services like Upload Thing
  • Platforms don't communicate these limitations well enough

Netlify CEO Response

  • CEO responded on Hacker News after viral Reddit post (8,000 upvotes)
  • Policy: don't shut down free sites during traffic spikes
  • Forgive bills for legitimate mistakes after the fact
  • Acknowledged initial support reply failure
  • Full refund provided to affected developer

Vercel Similar Incidents

Mike Ramirez Recursive Call Issue

  • Astro app with recursive API calls to itself
  • Racked up massive bill at $500/hour
  • Could have cost $30K in few days
  • Initial support response: "you ran compute, you pay"
  • Speaker escalated but insufficient effort initially
  • Developer went public with experience
  • Vercel eventually refunded and implemented recursive call detection
  • Now prevents same endpoint calling itself recursively

Michael's Stripe Webhook Attack

  • 56,000 fake Stripe accounts created by spammer
  • Stripe fired massive webhook volume to Vercel service
  • Vercel bill hit $23,000
  • Issue: unhandled webhook event types caused 60-second timeouts
  • Each timeout triggered Stripe retry loop
  • Blame distribution: 80% Stripe, 10-15% developer, 5-10% Vercel
  • Vercel CEO (Guillermo) provided full refund
  • Recommended reducing max duration timeouts, handling all endpoint responses

Platform Philosophy Debate

  • Companies don't want to charge individual developers huge amounts
  • $100K individual bill nothing compared to multi-million enterprise contracts
  • Bad optics hurt enterprise sales significantly
  • Goal isn't cash grab from accidental overspend
  • Spend limits problematic: site goes down at $101 if limit is $100
  • "Can't go back and fix ruined moment of glory, but can always cancel invoice"
  • Platforms designed to handle zero-to-viral traffic spikes

DDoS Protection Complexity

  • Can't reliably layer CloudFlare in front of Vercel/Netlify
  • Speaker experienced issues with cached HTML pointing to non-existent JS
  • Rate limiting Stripe endpoints would break legitimate business spikes
  • Platforms promise easy developer experience including protection
  • Responsibility falls on platforms to provide CloudFlare-level protection or better

Traffic Pattern Analysis

  • Netlify claimed traffic matched "organic traffic from region with lots of old devices"
  • Speaker skeptical - likely designed to appear organic by attacker
  • Developer requested IP information, request headers, connection data
  • Investigation ongoing but detailed forensics rarely available
  • Similar to speaker's 3-year ongoing DDoS situation - attackers remain unknown

Industry Response and Future Improvements

  • Vercel announced "coming soon: more ways to protect workloads from malicious traffic"
  • Platforms working toward not requiring manual rate limiters
  • Goal: reach beyond CloudFlare DDoS protection levels
  • Nothing is 100% secure in "chaotic world of hackers"
  • Focus on reasonable decisions with reasonable outcomes

Decision Framework for Platform Choice

  • Don't avoid platforms solely due to runaway bill stories (bills get refunded)
  • Valid reasons to avoid: cost multiplier too high for planned usage
  • Need DDoS mitigation strategy and budget if self-hosting
  • These incidents prove platforms handle mistakes better than alternatives
  • AWS typically doesn't refund unexpected bills like these platforms do

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome