AI summary by TwinMind
Reflection AI’s Misha Laskin on the AlphaGo Moment for LLMs | Training Data
Sequoia Capital
1h 7m 5s
Personal Background and Journey to AI
- Born in Russia, moved to Israel at age 1, then to Washington state at age 9
- Parents immigrated from Soviet Union with $300, pursued PhDs in chemistry at Hebrew University of Jerusalem
- Grew up in Tri-Cities, Washington - former Manhattan Project site where plutonium was enriched
- Discovered physics through Feynman lectures, pursued PhD in physics initially
- Realized physics was "root node problem" 100 years ago, not today
- Transitioned to startup work, then discovered deep learning and AlphaGo
- AlphaGo's "move 37" moment showed creative, superhuman agency - inspired career pivot to AI
- Joined Peter Abel's lab at Berkeley through persistence and cold outreach
- Eventually joined DeepMind to work on agents
AlphaGo and Early Agent Insights
- AlphaGo represented first large-scale superhuman agent
- Key insight: combining learning and search is optimal way to leverage compute
- Move 37 demonstrated creative problem-solving beyond brute force
- Field got "stuck" after AlphaGo - superhuman but narrow agents
- No clear path to generality due to data inefficiency (6 billion steps per task)
- Language models solved generality through internet as "massive multitask dataset"
- Internet contains many tasks: Wikipedia (historical description), Stack Overflow (coding Q&A)
- Current LLMs have generality but lack depth/reliability for agency
Gemini Experience and Post-Training Insights
- Collaborated closely with Yannis on Gemini
- Yannis led RLHF effort, Misha led reward model training
- Key insight: nothing specific about chat alignment - same approach works for any capability
- Architectures and models work well, bottlenecks being "washed away" with compute/scale
- Long context lengths solved faster than expected
- Training and alignment methods relatively stable
- Core challenge: data problem + enabling planning/search on top of models
Definition and Current State of AI Agents
- Agent definition: AI system that reasons independently and takes many steps to accomplish specified goals
- Goal specification evolved from reward functions (AlphaGo) to natural language
- Current "prompted agents" are on capability spectrum but low reliability
- Coding agents like SWE-agent, Devin achieving ~13-34% task completion rates
- Prompted agents represent heuristic approach that will be "washed away" by learning + search
- Prompts should specify goals, not put agents "on rails" or do thinking for them
The Breadth vs Depth Problem
- Current LLMs excel at breadth (multimodal, general capabilities)
- Lack depth in complex sequential reasoning/planning
- AlphaGo represents deepest agent ever built (one task only)
- Field needs to solve "depth problem" - someone must focus on this
- Large labs working on breadth, but depth problem needs dedicated focus
- Universal agents need both breadth AND depth
Technical Approach and Challenges
- Reinforcement learning from human feedback (RLHF) can work for agency
- Challenge: no ground truth rewards for most tasks (unlike games)
- Reward models are noisy and exploitable - policies find holes and exploit them
- Example: chatbot learning to never answer questions to avoid negative feedback
- "Scaling laws for reward model over-optimization" paper shows this happens at all scales
- Need better verification methods for task completion
- Recipe for agency: task sources, environments, algorithms, verification systems
Reflection AI's Mission and Vision
- Building universal superhuman agents with both breadth and depth
- Focus on general recipes that don't inherit task-specific heuristics
- Demonstrating across multiple environments: web agents, coding agents, OS agents
- Post-training as "hardening good behavior" - similar to AlphaGo's RL phase
- Timeline: ~3 years to something resembling digital AGI
- Urgency driven by AlphaGo-like acceleration potential in language models
Team and Hiring
- Attracted talent from top AI labs
- Yannis has "Michael Jordan effect" - inspires people to reach higher levels
- Looking for researchers and engineers methodically
- Common trait: hunger and urgency to solve the problem
- Belief that they can move faster outside large organizations
- Focus on "main quest" vs "side quest" within big companies
Future Vision and Impact
- Individual productivity/impact will dramatically increase
- Humans won't work less, but capacity for contribution will expand
- AI agents will handle tedious work, enabling more ambitious personal goals
- Moving beyond learning to use tools toward having AI agents accomplish goals
- Circular benefit: better AI agents will accelerate AI research itself
Broader AI Perspectives
- Most excited about mechanistic interpretability - "neuroscience of language models"
- AI field currently like late 1800s physics - empirical results without theoretical understanding
- Need more "science of AI" to understand scaling laws, data mixing effects
- Admires Peter Abel (operational efficiency), Vlad Mnih (creativity + humility), Yannis (inspirational excellence)
- Advice for AI founders: work on problems that internally matter to you, independent of external factors
