AI summary by TwinMind
Things That Don't Scale, The Software Edition – Dalton Caldwell and Michael Seibel
Y Combinator
25m 56s
Introduction & 90/10 Solution Concept
- Michael Cybel and Dalton Caldwell discuss "do things that don't scale" for software companies
- Paul Buchheit (PB) coined "90/10 solution" - get 90% benefit for 10% work
- PB's credibility: invented Gmail (used by 1.5 billion people) as side project at Google
- Founders initially hate this approach but it's essential for early success
- Analogy: house with broken pipes - turn water on to find leaks vs. checking every pipe preemptively
Gmail Origin Story - Paul Buchheit's Hack
- PB frustrated with existing email product at Google
- First Gmail version: hacked email into Google Groups newsletter UI
- Eureka moment: could read own email in this interface, stopped using old client
- Built features incrementally as needed (writing emails came later)
- Co-worker feedback: "loads fast, great, but has your email - I want mine"
- Spread throughout Google organically
- Critical failure: Gmail down all morning, PB came late to work
- Hardware issue: corrupted hard drive, PB fixed with screwdriver in server room
- PB's realization: "that day I learned people really think email's important"
- Motivation: wanted Linux desktop, didn't want Outlook on Windows
- Google initially hesitant to release due to hardware concerns
- Gmail invite system wasn't growth hack - server capacity limitation forced invite-only model
Facebook Early Scaling Architecture
- Pre-AWS era: startups bought servers, placed in data centers
- Site crashes required physical trips to reboot servers (2am drives to Santa Clara)
- Dalton's company iMeme shared data center with early Facebook
- Facebook's neighbor setup: 8 janky super micro servers initially
- Each server labeled with university name (Stanford, Harvard, etc.)
- Scaling method: completely separate PHP instance per school
- Separate MySQL server per school, separate memcache per school
- URL structure: harvard.thefacebook.com, stanford.facebook.com
- Cross-school communication added later via bounce code
- Took years to build global user table and eliminate this hack
- Hardware/MySQL limitations made single database impossible at scale
Twitch/Justin.TV Video Streaming Challenges
- Live video traffic peaks 20x normal (vs 2-4x for regular websites)
- Can't test 20x peaks - must experience and fix in real-time
- Celebrity streams caused massive sudden traffic spikes
Static Page Hack
- Dynamic features (username, view count) destroyed servers during traffic spikes
- Solution: button to make any page completely static
- Disabled all dynamic features during high traffic
- Only video player and chat worked (separate systems)
- Later: Kyle and Emmitt developed partial page caching
Video Server Propagation Issue
- Popular streams needed multiple video servers for 100k+ viewers
- Algorithm worked for gradual growth, failed for instant celebrity traffic
- Problem: video system didn't know pre-stream audience size
- Solution: website communicated waiting audience to video system
- Pre-populated streams to required servers before going live
- Celebrities thought they were live but stream propagated first (few seconds delay)
iMeme Music Player Innovation
- Goal: play music in browser without launching external Real Player
- YouTube innovation: video in Flash player within browser
- iMeme hack: used video tooling for music
- Every music file converted to .flv (Flash video)
- Zero-bit video field, audio-only content
- Entire music site was technically video site with no video
- Leveraged existing open-source video tools instead of building music-specific ones
Twitch Bandwidth & Cost Solutions
Free Peering Hack
- Live video streaming extremely expensive
- Poor fundraising led to cash constraints
- Hired YouTube network ops expert
- Free peering: direct ISP connections, no middleman costs
- Some ISPs refused free peering arrangements
- Solution: after 10 minutes free viewing, blocked video with message
- Message: "Your ISP doesn't do free peering, call them to complain" + contact info
- Strategy worked - ISPs agreed to peering after user complaints
Community Translation
- Global audience, translation companies quoted "infinite money" for 40 languages
- Borrowed Reddit's approach: community-driven translation
- Built website for volunteer translators
- Served English strings to non-English users with translation prompt
- Alternative was no translation at all
- Later added multiple translator verification system
- Got full product translation for free
Google Search Index Crisis (Secondhand Account)
- Original PageRank algorithm worked well for small web
- Internet growth outpaced hardware scaling ability (pre-cloud era)
- Big batch process to re-index web started failing (~2001)
- Process took 3 weeks to run, kept crashing
- For 3-4 months: no new Google index updates
- Users received stale results, new websites not indexed
- Google didn't shut down - users unaware of staleness
- Crisis led to MapReduce development (paper published)
- MapReduce enabled parallelized crawling/indexing
- Spawned Hadoop and modern big data tools
- Every major internet company uses descendants of this technology
Key Principles & Lessons
- "Turn the water on" approach: launch and fix what breaks vs. pre-optimizing
- Earn privilege to work on scalable solutions by first making something people want
- Apple example: Wozniak hand-soldering → AirPods (same company, different scales)
- Google earned right to build scalable tools by first having great search
- Don't make perfect the enemy of good
- Focus on giving users what they want, solve scaling problems afterward
- Best product decisions often made under duress (8pm, site down scenarios)
