AI summary by TwinMind

The Engineering Unlocks Behind DeepSeek | YC Decoded

Y Combinator

13m 5s

DeepSeek R1 Model Release and Market Impact

  • Chinese AI company DeepSeek announced R1 open source reasoning model
  • Claims comparable performance to OpenAI O1 at fraction of cost
  • Announcement triggered social media panic and stock market chaos
  • NVIDIA lost nearly $600 billion in market cap in single day
  • DeepSeek has been publishing research and releasing model weights for months
    • Following path similar to Meta's Llama model
    • Contrast to closed-weight competitors (OpenAI, Google DeepMind, Anthropic)
  • Broader public attention is what changed, not the technology itself

Model Architecture: DeepSeek V3 vs R1

  • Two distinct models: DeepSeek R1 and DeepSeek V3
  • DeepSeek V3 (released December 2024)
    • General purpose base model
    • Comparable performance to GPT-4o, Claude 3.5 Sonnet, Gemini 1.5
  • DeepSeek R1 (released end of January 2025)
    • Reasoning model built on top of V3
    • Applied algorithmic improvements to optimize reasoning ability
    • Comparable performance to OpenAI O1 and Google Flash 2.0 on complex reasoning benchmarks

Technical Innovations and Efficiency Optimizations

  • Many innovations discussed in V3 paper (December 2024), V2 paper (May 2024), and DeepSeek Math paper (February 2024)
  • V3 stitches together multiple innovations designed for compute and training efficiency

8-bit Floating Point Training

  • Trained natively in 8-bit floating point format vs usual 16-bit or 32-bit
  • Not new concept but key for massive memory savings without performance loss
  • FP8 accumulation fix: periodically merges calculations back into higher precision FP32 accumulator
    • Prevents small numerical errors from compounding
    • Results in far more efficient training across thousands of GPUs

GPU Utilization Challenges

  • AI labs typically see only 35% model FLOPS utilization (MFU)
  • GPUs idle most of time, waiting for data movement between caches/GPUs
  • NVIDIA's key advantage: integrated solution built over decade
    • Includes networking with InfiniBand, software with CUDA, developer experience
    • Lets researchers program GPU clusters as distributed system/"one giant GPU"

Mixture of Experts Architecture

  • DeepSeek V3: 671 billion total parameters, only 37 billion activated per token
  • Contrast: Llama 3 largest model activates full 405 billion parameters per token
  • V3 activates 11x fewer parameters per forward pass, saving massive computation
  • DeepSeek introduced novel techniques to stabilize performance and increase GPU utilization

Multi-Head Latent Attention (MLA)

  • First revealed in V2 paper (May 2024)
  • Tackles KV cache storage limitation (biggest RAM overhead source)
  • Compresses key/value matrices into latent representation, reconstructs when needed
  • V2 model achieved 93.3% KV cache size reduction
  • Boosted maximum generation throughput to 5.76x

Multi-Token Prediction (MTP)

  • Unlike traditional next-token prediction, V3 anticipates multiple future tokens per step
  • Densifies training signals, provides more feedback per step
  • Improves data efficiency and faster learning
  • Enables better representation planning for smoother, coherent outputs
  • MTP modules repurposed for speculative decoding, significantly speeding generation

Reasoning Model Development (R1)

  • LLMs can be improved by "think step by step" prompting
  • Reasoning models specifically trained to break down hard problems, think in paragraphs
  • OpenAI O1 (September 2024) achieved state-of-the-art results in math, coding, science
  • DeepSeek took similar approach, published "secret sauce"

Reinforcement Learning Approach

  • Both achieve results through reinforcement learning (RL)
  • Modern LLMs use RLHF (RL with human feedback) or RLAIF (RL from AI feedback)
  • Reasoning models apply RL specifically for step-by-step thinking through complex problems

DeepSeek's RL Implementation

  • Assembled problems with verifiable outputs (especially math/coding)
  • Designed training pipeline for model to think and output correct answers
  • No external examples of thinking provided (no human/AI examples)
  • Extremely simple grading process: simple rules for accuracy and formatting evaluation
  • Used Group Relative Policy Optimization (GRPO) technique (published February 2024)

R1-Zero Results

  • Reasoning emerged over thousands of RL steps
  • Model learned extended Chain of Thought skills
  • Experienced "aha moments" recognizing mistakes and backtracking
  • First large model achieving top-tier results purely through reinforcement learning
  • Similar to pure RL successes: DeepMind's AlphaGo (2016), OpenAI's Rubik's Cube robotics, Dota 2 victories

R1-Zero Limitations and R1 Improvements

  • R1-Zero suffered poor readability, random English/Chinese switching
  • DeepSeek introduced "cold start" phase: fine-tuning on structured reasoning examples before RL
  • Eliminated language mixing issues, made outputs comprehensible
  • R1 achieves comparable performance to O1 on math/coding benchmarks

Market Competition and Innovation Pace

  • 2 weeks after R1 release, OpenAI released O3 Mini
  • O3 Mini outperforms both R1 and O1 on key benchmarks
  • Innovation pace accelerating rapidly

Accessibility and Cost Advantages

  • R1 freely accessible through website/app
  • Free to download, run locally, and customize
  • Near state-of-the-art performance at fraction of price vs other reasoning models
  • Efficiency improvements enable broader accessibility

Training Cost Misconceptions

  • V3's alleged $5.5 million training cost widely misunderstood
  • $5.5 million refers only to final training run for V3
  • Excludes R1 training costs, R&D, hardware operating expenses
    • Presumably hundreds of millions given algorithmic optimizations
  • Work is reproducible: UC Berkeley lab applied R1-Zero techniques for $30

Industry Implications

  • Proves room for new players on AI frontier
  • Opportunities in: rebuilding stack, optimizing GPU workloads, improving inference software, developing AI-generated kernels
  • Fantastic news for AI applications (consumer/B2B): cost of intelligence decreasing
  • Best time to build AI startups

Y Combinator Promotion

  • YC spring batch application deadline: February 11th
  • $500,000 investment plus startup community access for accepted applicants

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome