AI summary by TwinMind
The Engineering Unlocks Behind DeepSeek | YC Decoded
Y Combinator
13m 5s
DeepSeek R1 Model Release and Market Impact
- Chinese AI company DeepSeek announced R1 open source reasoning model
- Claims comparable performance to OpenAI O1 at fraction of cost
- Announcement triggered social media panic and stock market chaos
- NVIDIA lost nearly $600 billion in market cap in single day
- DeepSeek has been publishing research and releasing model weights for months
- Following path similar to Meta's Llama model
- Contrast to closed-weight competitors (OpenAI, Google DeepMind, Anthropic)
- Broader public attention is what changed, not the technology itself
Model Architecture: DeepSeek V3 vs R1
- Two distinct models: DeepSeek R1 and DeepSeek V3
- DeepSeek V3 (released December 2024)
- General purpose base model
- Comparable performance to GPT-4o, Claude 3.5 Sonnet, Gemini 1.5
- DeepSeek R1 (released end of January 2025)
- Reasoning model built on top of V3
- Applied algorithmic improvements to optimize reasoning ability
- Comparable performance to OpenAI O1 and Google Flash 2.0 on complex reasoning benchmarks
Technical Innovations and Efficiency Optimizations
- Many innovations discussed in V3 paper (December 2024), V2 paper (May 2024), and DeepSeek Math paper (February 2024)
- V3 stitches together multiple innovations designed for compute and training efficiency
8-bit Floating Point Training
- Trained natively in 8-bit floating point format vs usual 16-bit or 32-bit
- Not new concept but key for massive memory savings without performance loss
- FP8 accumulation fix: periodically merges calculations back into higher precision FP32 accumulator
- Prevents small numerical errors from compounding
- Results in far more efficient training across thousands of GPUs
GPU Utilization Challenges
- AI labs typically see only 35% model FLOPS utilization (MFU)
- GPUs idle most of time, waiting for data movement between caches/GPUs
- NVIDIA's key advantage: integrated solution built over decade
- Includes networking with InfiniBand, software with CUDA, developer experience
- Lets researchers program GPU clusters as distributed system/"one giant GPU"
Mixture of Experts Architecture
- DeepSeek V3: 671 billion total parameters, only 37 billion activated per token
- Contrast: Llama 3 largest model activates full 405 billion parameters per token
- V3 activates 11x fewer parameters per forward pass, saving massive computation
- DeepSeek introduced novel techniques to stabilize performance and increase GPU utilization
Multi-Head Latent Attention (MLA)
- First revealed in V2 paper (May 2024)
- Tackles KV cache storage limitation (biggest RAM overhead source)
- Compresses key/value matrices into latent representation, reconstructs when needed
- V2 model achieved 93.3% KV cache size reduction
- Boosted maximum generation throughput to 5.76x
Multi-Token Prediction (MTP)
- Unlike traditional next-token prediction, V3 anticipates multiple future tokens per step
- Densifies training signals, provides more feedback per step
- Improves data efficiency and faster learning
- Enables better representation planning for smoother, coherent outputs
- MTP modules repurposed for speculative decoding, significantly speeding generation
Reasoning Model Development (R1)
- LLMs can be improved by "think step by step" prompting
- Reasoning models specifically trained to break down hard problems, think in paragraphs
- OpenAI O1 (September 2024) achieved state-of-the-art results in math, coding, science
- DeepSeek took similar approach, published "secret sauce"
Reinforcement Learning Approach
- Both achieve results through reinforcement learning (RL)
- Modern LLMs use RLHF (RL with human feedback) or RLAIF (RL from AI feedback)
- Reasoning models apply RL specifically for step-by-step thinking through complex problems
DeepSeek's RL Implementation
- Assembled problems with verifiable outputs (especially math/coding)
- Designed training pipeline for model to think and output correct answers
- No external examples of thinking provided (no human/AI examples)
- Extremely simple grading process: simple rules for accuracy and formatting evaluation
- Used Group Relative Policy Optimization (GRPO) technique (published February 2024)
R1-Zero Results
- Reasoning emerged over thousands of RL steps
- Model learned extended Chain of Thought skills
- Experienced "aha moments" recognizing mistakes and backtracking
- First large model achieving top-tier results purely through reinforcement learning
- Similar to pure RL successes: DeepMind's AlphaGo (2016), OpenAI's Rubik's Cube robotics, Dota 2 victories
R1-Zero Limitations and R1 Improvements
- R1-Zero suffered poor readability, random English/Chinese switching
- DeepSeek introduced "cold start" phase: fine-tuning on structured reasoning examples before RL
- Eliminated language mixing issues, made outputs comprehensible
- R1 achieves comparable performance to O1 on math/coding benchmarks
Market Competition and Innovation Pace
- 2 weeks after R1 release, OpenAI released O3 Mini
- O3 Mini outperforms both R1 and O1 on key benchmarks
- Innovation pace accelerating rapidly
Accessibility and Cost Advantages
- R1 freely accessible through website/app
- Free to download, run locally, and customize
- Near state-of-the-art performance at fraction of price vs other reasoning models
- Efficiency improvements enable broader accessibility
Training Cost Misconceptions
- V3's alleged $5.5 million training cost widely misunderstood
- $5.5 million refers only to final training run for V3
- Excludes R1 training costs, R&D, hardware operating expenses
- Presumably hundreds of millions given algorithmic optimizations
- Work is reproducible: UC Berkeley lab applied R1-Zero techniques for $30
Industry Implications
- Proves room for new players on AI frontier
- Opportunities in: rebuilding stack, optimizing GPU workloads, improving inference software, developing AI-generated kernels
- Fantastic news for AI applications (consumer/B2B): cost of intelligence decreasing
- Best time to build AI startups
Y Combinator Promotion
- YC spring batch application deadline: February 11th
- $500,000 investment plus startup community access for accepted applicants
