AI summary by TwinMind
John Schulman (OpenAI Cofounder) - Reasoning, RLHF, & Plan for 2027 AGI
Dwarkesh Patel
1h 36m 54s
Pre-Training vs Post-Training Fundamentals
- Pre-training: models learn to imitate all internet content (websites, code)
- Trained to maximize likelihood, predict next token
- Creates calibrated models that can generate diverse content and assign probabilities
- Base model can take on different personas, generate various content types
- Post-training: targets narrower behavior range
- Shapes model to behave like chat assistant with specific persona
- Optimizes for outputs humans find useful vs raw web content imitation
- Creates helpful, question-answering behavior rather than person imitation
Future AI Capabilities Timeline (1-5 Years)
- 1-2 years: models capable of whole coding projects
- Move from single function suggestions to multi-file projects
- High-level instructions → model writes/tests/iterates multiple files
- Fundamental unlock: acting coherently for extended periods
- Capability improvements through:
- Training on harder, longer-horizon tasks (currently mostly single-step data)
- RL training for multi-step projects
- Better error recovery and edge case handling
- More sample-efficient learning from limited data
Long-Horizon RL and Scaling Challenges
- Current models act coherently ~5 minutes, need hour/week/month-long tasks
- Scaling may not follow clean 10x compute law per benchmark
- Phase transitions possible where same mechanisms work across time scales
- Humans use similar mental machinery for planning month vs year vs decade ahead
- Language enables describing all time scales, planning progress toward distant goals
Potential Bottlenecks Beyond Long-Horizon Capability
- Human expert experience: taste, ambiguity handling
- Research-specific considerations beyond pure task execution
- Physical world affordances and UI interactions
- Mundane barriers (website compatibility, multimodal interfaces)
- Models will likely use human-designed UIs initially
- Some websites may optimize for AI interaction (better text representations)
Generalization Evidence in Post-Training
- English fine-tuning automatically improves other languages
- Text-only fine-tuning improves multimodal behavior
- 30 examples of "model limitations" generalized to many uncovered capabilities
- Small amounts of targeted data can fix broad behavioral issues
- Models show strong transfer between related domains
AGI Timeline and Safety Considerations
- Schulman: reasonable to plan for AGI possibility soon, though not expecting next year
- If AGI arrives unexpectedly early:
- Slow down training/deployment until safety understanding improves
- Careful sandboxing of smarter systems
- Coordination needed among major AI entities to avoid race dynamics
- Preferred scenario: gradual capability increases with corresponding safety improvements
- Avoid discontinuous jumps that create coordination challenges
Safety and Alignment Approaches
- Defense in depth strategy:
- Well-behaved models with strong moral compass
- Resistance to takeover/severe misuse
- Comprehensive monitoring systems
- Simulated deployment testing (red teaming)
- Current RLHF feels safe: models only optimize for human-approved text
- Long-horizon RL introduces new risks: models taking unusual actions during task execution
- Instrumental convergence concerns for poorly specified tasks
Current RLHF System Characteristics
- Functions like drive/goal in human psychology
- Models trained to maximize human approval via reward model
- Creates consistent "helpful assistant" persona
- Results in verbose, bullet-point heavy, formal responses
- May be due to: training on single messages, labeler preferences, speed considerations
- Unintentional distillation between providers (contractors using other chatbots)
Reasoning and Test-Time Computation
- Two approaches for better reasoning:
- Training-time: learn to follow correct reasoning chains
- Test-time: use compute during inference for step-by-step reasoning
- Best results likely combine both approaches
- Reasoning requires test-time computation by definition
- Models need practice (training-time computation) to improve
Memory and Learning Systems
- Gap between pre-training (trillions of tokens) and in-context learning
- Need for medium-term memory: more than context, less than pre-training scale
- Active learning: models understanding knowledge gaps, seeking relevant information
- Introspection capabilities could enable self-directed learning
- Long context windows (100K-1M tokens) reducing need for alternative approaches
- Online learning + cognitive introspection skills likely complementary
Research and Development Insights
- GPT-4 ELO score improved 100 points since release, mostly from post-training
- Multiple improvement axes: data quality, quantity, process iterations
- Post-training creates significant competitive moat due to complexity
- Requires skilled teams, tacit knowledge, organizational expertise
- ML field generally healthy compared to social sciences due to practical grounding
ChatGPT Development History
- Built on GPT-3.5 (finished training early 2022)
- Evolved from instruction-following models and web browsing research
- Chat format more intuitive than instruction completion for users and labelers
- Required iterative supervised fine-tuning, not just one-shot human data training
- Mixed instruction and chat datasets for best-of-both-worlds approach
Technical Scaling and Efficiency
- Bigger models more sample-efficient: learn same capability with less data
- Possible explanations: ensemble of parallel circuits, mixture of experts concept
- Bigger models have larger "library" of computational circuits
- No clear understanding of why parameter scaling works so well
- Continued efficiency improvements in both pre-training and post-training
Data and Training Considerations
- Approaching but not immediately hitting data wall
- Pre-training nature will change as high-quality data becomes scarce
- Post-training compute allocation likely to increase significantly
- Model-generated content higher quality than much web content
- Synthetic data and model self-improvement becoming more important
Deployment and Integration
- AI still small fraction of economy, many jobs can't be helped yet
- Growth from both capability improvements and better integration
- Expect widespread adoption across more sophisticated tasks
- Science acceleration through literature comprehension and data analysis
- Form factor evolution: from search engine to collaborative project partner
- Proactive suggestions and background work rather than one-off queries
Stakeholder Alignment Framework
- Four stakeholder categories: end user, developer, platform (OpenAI), humanity
- Model spec addresses conflicting stakeholder demands
- Generally follow user instructions while preventing harm to others
- Avoid paternalistic behavior, maintain neutrality on opinions
- Balance helpfulness with harm prevention
Future Concerns and Considerations
- Economic pressures may force removal of human oversight
- Amdahl's law: human bottlenecks limit firm efficiency gains
- Regulation challenges across companies and countries
- Need coordination among model providers or universal regulatory regimes
- AI-run firms may have higher tail risks despite better average performance
