AI summary by TwinMind
Why GPT-4 is much smarter than it was a year ago – OpenAI cofounder John Schulman
Dwarkesh Patel
5m 4s
Compute Allocation Shift: Pre-training vs Post-training
- Current ratio heavily lopsided toward pre-training
- Expected shift toward more post-training compute allocation
- Rationale: model output quality now higher than most web content
- Makes more sense for model to "think by itself" rather than imitate web content
- First principles argument supports this approach
- Significant gains found through post-training methodology
- Plan to continue pushing this methodology and increase compute allocation
GPT-4 Performance Improvements
- Current GPT-4 ELO score: 100 points higher than original release
- Most improvement attributed to post-training enhancements
- Multiple improvement axes:
- Data quality improvements
- Data quantity increases
- More iterations of deployment/data collection process
- Changing annotation collection types
- Combined improvements create effective compute increase
Competitive Moats in Post-training
- Complex operation requiring skilled personnel
- Significant tacit knowledge required
- Organizational knowledge accumulation necessary
- Complicated effort to create models with desired functionality
- Accumulation of substantial R&D investment
- Not trivial to spin up immediately
- Same companies doing serious pre-training also doing serious post-training
- Somewhat possible to copy/spin up efforts
Circumventing Moats
- Model distillation possible
- Can clone outputs from other models
- Can use other models as judges for comparisons
- Big league players avoid this due to:
- Terms of service violations
- Pride considerations
- Smaller players expected to use these methods to get started
Skills for Effective Post-training Research
- Experience across different parts of the stack:
- RL algorithms (from grad school background)
- Data collection processes
- Annotation processes
- Language model experimentation
- Holistic view of entire stack required
- High curiosity about different components
- Balanced approach:
- Empirical methodology using experiments to update views
- First principles thinking about ideal data collection
- Assumption that learning works as foundation
