AI summary by TwinMind

Reflection AI’s Misha Laskin on the AlphaGo Moment for LLMs | Training Data

Sequoia Capital

1h 7m 5s

Personal Background and Journey to AI

  • Born in Russia, moved to Israel at age 1, then to Washington state at age 9
  • Parents immigrated from Soviet Union with $300, pursued PhDs in chemistry at Hebrew University of Jerusalem
  • Grew up in Tri-Cities, Washington - former Manhattan Project site where plutonium was enriched
  • Discovered physics through Feynman lectures, pursued PhD in physics initially
  • Realized physics was "root node problem" 100 years ago, not today
  • Transitioned to startup work, then discovered deep learning and AlphaGo
  • AlphaGo's "move 37" moment showed creative, superhuman agency - inspired career pivot to AI
  • Joined Peter Abel's lab at Berkeley through persistence and cold outreach
  • Eventually joined DeepMind to work on agents

AlphaGo and Early Agent Insights

  • AlphaGo represented first large-scale superhuman agent
  • Key insight: combining learning and search is optimal way to leverage compute
  • Move 37 demonstrated creative problem-solving beyond brute force
  • Field got "stuck" after AlphaGo - superhuman but narrow agents
  • No clear path to generality due to data inefficiency (6 billion steps per task)
  • Language models solved generality through internet as "massive multitask dataset"
  • Internet contains many tasks: Wikipedia (historical description), Stack Overflow (coding Q&A)
  • Current LLMs have generality but lack depth/reliability for agency

Gemini Experience and Post-Training Insights

  • Collaborated closely with Yannis on Gemini
  • Yannis led RLHF effort, Misha led reward model training
  • Key insight: nothing specific about chat alignment - same approach works for any capability
  • Architectures and models work well, bottlenecks being "washed away" with compute/scale
  • Long context lengths solved faster than expected
  • Training and alignment methods relatively stable
  • Core challenge: data problem + enabling planning/search on top of models

Definition and Current State of AI Agents

  • Agent definition: AI system that reasons independently and takes many steps to accomplish specified goals
  • Goal specification evolved from reward functions (AlphaGo) to natural language
  • Current "prompted agents" are on capability spectrum but low reliability
  • Coding agents like SWE-agent, Devin achieving ~13-34% task completion rates
  • Prompted agents represent heuristic approach that will be "washed away" by learning + search
  • Prompts should specify goals, not put agents "on rails" or do thinking for them

The Breadth vs Depth Problem

  • Current LLMs excel at breadth (multimodal, general capabilities)
  • Lack depth in complex sequential reasoning/planning
  • AlphaGo represents deepest agent ever built (one task only)
  • Field needs to solve "depth problem" - someone must focus on this
  • Large labs working on breadth, but depth problem needs dedicated focus
  • Universal agents need both breadth AND depth

Technical Approach and Challenges

  • Reinforcement learning from human feedback (RLHF) can work for agency
  • Challenge: no ground truth rewards for most tasks (unlike games)
  • Reward models are noisy and exploitable - policies find holes and exploit them
  • Example: chatbot learning to never answer questions to avoid negative feedback
  • "Scaling laws for reward model over-optimization" paper shows this happens at all scales
  • Need better verification methods for task completion
  • Recipe for agency: task sources, environments, algorithms, verification systems

Reflection AI's Mission and Vision

  • Building universal superhuman agents with both breadth and depth
  • Focus on general recipes that don't inherit task-specific heuristics
  • Demonstrating across multiple environments: web agents, coding agents, OS agents
  • Post-training as "hardening good behavior" - similar to AlphaGo's RL phase
  • Timeline: ~3 years to something resembling digital AGI
  • Urgency driven by AlphaGo-like acceleration potential in language models

Team and Hiring

  • Attracted talent from top AI labs
  • Yannis has "Michael Jordan effect" - inspires people to reach higher levels
  • Looking for researchers and engineers methodically
  • Common trait: hunger and urgency to solve the problem
  • Belief that they can move faster outside large organizations
  • Focus on "main quest" vs "side quest" within big companies

Future Vision and Impact

  • Individual productivity/impact will dramatically increase
  • Humans won't work less, but capacity for contribution will expand
  • AI agents will handle tedious work, enabling more ambitious personal goals
  • Moving beyond learning to use tools toward having AI agents accomplish goals
  • Circular benefit: better AI agents will accelerate AI research itself

Broader AI Perspectives

  • Most excited about mechanistic interpretability - "neuroscience of language models"
  • AI field currently like late 1800s physics - empirical results without theoretical understanding
  • Need more "science of AI" to understand scaling laws, data mixing effects
  • Admires Peter Abel (operational efficiency), Vlad Mnih (creativity + humility), Yannis (inspirational excellence)
  • Advice for AI founders: work on problems that internally matter to you, independent of external factors

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome