AI summary by TwinMind
OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better
Sequoia Capital
45m 22s
Introduction and Team Background
- Interview with Noam, Hunter, and Ilya from OpenAI's O1 (Project Strawberry) team
- O1 represents OpenAI's first major foray into general inference-time compute
- Discussion covers reasoning, chain of thought, and inference-time scaling laws
Reasoning Definition and System 1 vs System 2 Thinking
- Reasoning defined as problems benefiting from longer thinking time
- System 1: automatic, instinctive responses
- System 2: slower, process-driven responses
- Examples:
- Capital of Bhutan: no benefit from extended thinking (factual recall)
- Sudoku puzzle: clear benefit from longer consideration
- Generator-verifier gap: hard to generate solutions but easy to verify correctness
- Problems exist on spectrum from easy-to-verify to equally hard to verify/generate
Team Conviction and Development Journey
- Initial skepticism from some team members about approach viability
- Years of research with many failed attempts before breakthrough
- OpenAI leadership maintained conviction despite setbacks
- Ilya's background: 5.5 years at OpenAI, initially focused on robotics
- Empirical, data-driven approach guided development decisions
Technical Approach and Architecture
- O1 models trained with reinforcement learning (RL) to "think" and reason
- Fundamentally different from traditional language models
- Generalizes across multiple reasoning domains
- Uses text interface for broad adaptability across problem types
Breakthrough Moments and Backtracking Discovery
- Key breakthrough: observing model exhibit "backtracking" behavior
- Model would recognize mistakes: "wait this is wrong, let me take a step back"
- Surprised researchers as autoregressive models typically just predict next token
- Higher performance on math evaluations than previous bespoke systems
- Emergent abilities developed from allowing longer thinking time
AlphaGo Parallels and Game AI Lessons
- AlphaGo took 30 seconds per move, performed worse when forced to act instantly
- Monte Carlo Tree Search worked well for Go but not other domains like poker
- O1's reasoning approach is more general than domain-specific methods
- Can apply reinforcement learning to general text interface
Real-World Applications and Surprising Use Cases
- Medical researchers using O1 as brainstorming partner for cancer research
- Gene discovery and therapy applications
- Model cannot conduct research itself but serves as effective collaborator
- Programming competition performance with unconventional solution approaches
- Geometry problem solving with visualization techniques
Deep RL Renaissance
- Deep RL experienced "trough of disillusionment" after initial hype (2015-2019)
- GPT paradigm initially flew under radar except at OpenAI
- O1 demonstrates deep RL's power when combined with other elements
- First major general application of deep RL beyond narrow game settings
Performance Characteristics and Limitations
- Excels at STEM subjects due to reasoning-friendly problem structure
- Passes OpenAI research engineer interviews with high success rates
- Already authored pull requests in OpenAI's codebase
- Significant gaps remain in humanities and creative domains
- ~120 IQ on some tests, below 40 on creative writing tasks
Chain of Thought and Transparency
- Decision to hide chain of thought reasoning was not contentious
- Similar rationale to not sharing model weights for frontier models
- Security and competitive reasons for concealment
- Chain of thought example: solving integral with step-by-step simplification process
Inference-Time Scaling Laws
- Most significant result: scaling laws for inference-time compute
- Addresses concerns about AI hitting walls due to pre-training costs
- New dimension for scaling previously untapped
- Implications for future AI development trajectory
Engineering Challenges and Bottlenecks
- Scaling inference-time compute involves significant engineering challenges
- Similar complexity to scaling from GPT-2 to GPT-3 to GPT-4
- Need to identify appropriate evaluation benchmarks
- Academic researchers often surprised by engineering vs research problem ratio
Future Implications and Theoretical Limits
- Theoretical possibility of solving any math theorem with infinite compute
- Diminishing returns expected with longer thinking times
- Comparison to Asimov's "The Last Question" - AI thinking for millennia
- Practical applications likely in math research assistance and software engineering
Model Naming and Misconceptions
- "Strawberry" codename coincidentally aligned with popular reasoning question
- Name actually came from someone eating strawberries during naming meeting
- Not related to "how many R's in strawberry" reasoning test
- Generally well-understood by community upon release
O1 Mini and Model Variants
- O1 Mini: faster, cheaper version for different use cases
- Enables faster iteration for researchers and developers
- Different models serve different purposes in ecosystem
- Tradeoff between speed/cost and capability
Future Development and Feedback
- Team not running out of ideas for future versions
- Eager for user feedback to identify new domains and applications
- Bias toward domains team understands vs. broader use cases
- Trend line from scaling laws expected to continue extending
