AI summary by TwinMind

Things That Don't Scale, The Software Edition – Dalton Caldwell and Michael Seibel

Y Combinator

25m 56s

Introduction & 90/10 Solution Concept

  • Michael Cybel and Dalton Caldwell discuss "do things that don't scale" for software companies
  • Paul Buchheit (PB) coined "90/10 solution" - get 90% benefit for 10% work
  • PB's credibility: invented Gmail (used by 1.5 billion people) as side project at Google
  • Founders initially hate this approach but it's essential for early success
  • Analogy: house with broken pipes - turn water on to find leaks vs. checking every pipe preemptively

Gmail Origin Story - Paul Buchheit's Hack

  • PB frustrated with existing email product at Google
  • First Gmail version: hacked email into Google Groups newsletter UI
  • Eureka moment: could read own email in this interface, stopped using old client
  • Built features incrementally as needed (writing emails came later)
  • Co-worker feedback: "loads fast, great, but has your email - I want mine"
  • Spread throughout Google organically
  • Critical failure: Gmail down all morning, PB came late to work
  • Hardware issue: corrupted hard drive, PB fixed with screwdriver in server room
  • PB's realization: "that day I learned people really think email's important"
  • Motivation: wanted Linux desktop, didn't want Outlook on Windows
  • Google initially hesitant to release due to hardware concerns
  • Gmail invite system wasn't growth hack - server capacity limitation forced invite-only model

Facebook Early Scaling Architecture

  • Pre-AWS era: startups bought servers, placed in data centers
  • Site crashes required physical trips to reboot servers (2am drives to Santa Clara)
  • Dalton's company iMeme shared data center with early Facebook
  • Facebook's neighbor setup: 8 janky super micro servers initially
  • Each server labeled with university name (Stanford, Harvard, etc.)
  • Scaling method: completely separate PHP instance per school
  • Separate MySQL server per school, separate memcache per school
  • URL structure: harvard.thefacebook.com, stanford.facebook.com
  • Cross-school communication added later via bounce code
  • Took years to build global user table and eliminate this hack
  • Hardware/MySQL limitations made single database impossible at scale

Twitch/Justin.TV Video Streaming Challenges

  • Live video traffic peaks 20x normal (vs 2-4x for regular websites)
  • Can't test 20x peaks - must experience and fix in real-time
  • Celebrity streams caused massive sudden traffic spikes

Static Page Hack

  • Dynamic features (username, view count) destroyed servers during traffic spikes
  • Solution: button to make any page completely static
  • Disabled all dynamic features during high traffic
  • Only video player and chat worked (separate systems)
  • Later: Kyle and Emmitt developed partial page caching

Video Server Propagation Issue

  • Popular streams needed multiple video servers for 100k+ viewers
  • Algorithm worked for gradual growth, failed for instant celebrity traffic
  • Problem: video system didn't know pre-stream audience size
  • Solution: website communicated waiting audience to video system
  • Pre-populated streams to required servers before going live
  • Celebrities thought they were live but stream propagated first (few seconds delay)

iMeme Music Player Innovation

  • Goal: play music in browser without launching external Real Player
  • YouTube innovation: video in Flash player within browser
  • iMeme hack: used video tooling for music
  • Every music file converted to .flv (Flash video)
  • Zero-bit video field, audio-only content
  • Entire music site was technically video site with no video
  • Leveraged existing open-source video tools instead of building music-specific ones

Twitch Bandwidth & Cost Solutions

Free Peering Hack

  • Live video streaming extremely expensive
  • Poor fundraising led to cash constraints
  • Hired YouTube network ops expert
  • Free peering: direct ISP connections, no middleman costs
  • Some ISPs refused free peering arrangements
  • Solution: after 10 minutes free viewing, blocked video with message
  • Message: "Your ISP doesn't do free peering, call them to complain" + contact info
  • Strategy worked - ISPs agreed to peering after user complaints

Community Translation

  • Global audience, translation companies quoted "infinite money" for 40 languages
  • Borrowed Reddit's approach: community-driven translation
  • Built website for volunteer translators
  • Served English strings to non-English users with translation prompt
  • Alternative was no translation at all
  • Later added multiple translator verification system
  • Got full product translation for free

Google Search Index Crisis (Secondhand Account)

  • Original PageRank algorithm worked well for small web
  • Internet growth outpaced hardware scaling ability (pre-cloud era)
  • Big batch process to re-index web started failing (~2001)
  • Process took 3 weeks to run, kept crashing
  • For 3-4 months: no new Google index updates
  • Users received stale results, new websites not indexed
  • Google didn't shut down - users unaware of staleness
  • Crisis led to MapReduce development (paper published)
  • MapReduce enabled parallelized crawling/indexing
  • Spawned Hadoop and modern big data tools
  • Every major internet company uses descendants of this technology

Key Principles & Lessons

  • "Turn the water on" approach: launch and fix what breaks vs. pre-optimizing
  • Earn privilege to work on scalable solutions by first making something people want
  • Apple example: Wozniak hand-soldering → AirPods (same company, different scales)
  • Google earned right to build scalable tools by first having great search
  • Don't make perfect the enemy of good
  • Focus on giving users what they want, solve scaling problems afterward
  • Best product decisions often made under duress (8pm, site down scenarios)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome