AI summary by TwinMind
Surprise $104,500 Bill? Breaking Down A BIG Mistake
Theo - t3․gg
27m 11s
Initial Billing Incident
- Developer received $104,500 overdue bill from Netlify for simple static site
- Initially assumed phishing scheme but was legitimate
- Site had insane bandwidth usage totaling $114,000
- Peak day: 60.7 terabytes traffic in single day
- Site normally stayed within free tier, never exceeded 10GB monthly
- Only 200 daily visitors typically
- Attack focused on 3.4MB sound file hosted on site
Netlify's Initial Support Response Failure
- Default offer: charge 20% of bill
- "Generous" offer: only charge 5% = $5,000
- Speaker calls this "absurd" and "incredibly unacceptable"
- $5,000 bill for hobby project due to external abuse should not happen
- Support response was biggest failure by company
Platform Protection Analysis
- Platforms do have DDoS protection, not completely defenseless
- Similar incident happened to speaker ~1.5-2 years ago on Vercel
- Attackers targeted largest file (half megabyte JS bundle)
- Vercel used opportunity to improve servers and filters
- Speaker didn't pay anything due to relationship + public attention
Notification System Failures
- Developer should have been alerted when spending skyrocketed
- Claims no notifications received, only "extra usage package purchased" email
- Speaker gets daily Vercel notifications for limit breaches
- Balance needed: too many notifications train users to ignore them
- 20%+ billing increases should trigger automatic notifications
- Lack of notifications alone justifies bill forgiveness
File Hosting Best Practices
- Large files shouldn't be hosted on Vercel/Netlify due to bandwidth costs
- These platforms optimize for performance via global CDN
- 3.4MB MP3 file inappropriate for these services
- Alternatives: SoundCloud, GitHub assets, upload services like Upload Thing
- Platforms don't communicate these limitations well enough
Netlify CEO Response
- CEO responded on Hacker News after viral Reddit post (8,000 upvotes)
- Policy: don't shut down free sites during traffic spikes
- Forgive bills for legitimate mistakes after the fact
- Acknowledged initial support reply failure
- Full refund provided to affected developer
Vercel Similar Incidents
Mike Ramirez Recursive Call Issue
- Astro app with recursive API calls to itself
- Racked up massive bill at $500/hour
- Could have cost $30K in few days
- Initial support response: "you ran compute, you pay"
- Speaker escalated but insufficient effort initially
- Developer went public with experience
- Vercel eventually refunded and implemented recursive call detection
- Now prevents same endpoint calling itself recursively
Michael's Stripe Webhook Attack
- 56,000 fake Stripe accounts created by spammer
- Stripe fired massive webhook volume to Vercel service
- Vercel bill hit $23,000
- Issue: unhandled webhook event types caused 60-second timeouts
- Each timeout triggered Stripe retry loop
- Blame distribution: 80% Stripe, 10-15% developer, 5-10% Vercel
- Vercel CEO (Guillermo) provided full refund
- Recommended reducing max duration timeouts, handling all endpoint responses
Platform Philosophy Debate
- Companies don't want to charge individual developers huge amounts
- $100K individual bill nothing compared to multi-million enterprise contracts
- Bad optics hurt enterprise sales significantly
- Goal isn't cash grab from accidental overspend
- Spend limits problematic: site goes down at $101 if limit is $100
- "Can't go back and fix ruined moment of glory, but can always cancel invoice"
- Platforms designed to handle zero-to-viral traffic spikes
DDoS Protection Complexity
- Can't reliably layer CloudFlare in front of Vercel/Netlify
- Speaker experienced issues with cached HTML pointing to non-existent JS
- Rate limiting Stripe endpoints would break legitimate business spikes
- Platforms promise easy developer experience including protection
- Responsibility falls on platforms to provide CloudFlare-level protection or better
Traffic Pattern Analysis
- Netlify claimed traffic matched "organic traffic from region with lots of old devices"
- Speaker skeptical - likely designed to appear organic by attacker
- Developer requested IP information, request headers, connection data
- Investigation ongoing but detailed forensics rarely available
- Similar to speaker's 3-year ongoing DDoS situation - attackers remain unknown
Industry Response and Future Improvements
- Vercel announced "coming soon: more ways to protect workloads from malicious traffic"
- Platforms working toward not requiring manual rate limiters
- Goal: reach beyond CloudFlare DDoS protection levels
- Nothing is 100% secure in "chaotic world of hackers"
- Focus on reasonable decisions with reasonable outcomes
Decision Framework for Platform Choice
- Don't avoid platforms solely due to runaway bill stories (bills get refunded)
- Valid reasons to avoid: cost multiplier too high for planned usage
- Need DDoS mitigation strategy and budget if self-hosting
- These incidents prove platforms handle mistakes better than alternatives
- AWS typically doesn't refund unexpected bills like these platforms do
