AI summary by TwinMind
AI Scaling Hits Wall, Rumours Say. How Serious is it?
Sabine Hossenfelder
7m 26s
AI Industry Claims vs Reality
- Sam Altman (OpenAI) claims AI will solve all physics problems
- September essay "The Intelligence Age" - confident AI gets better with scale
- Predicts superintelligence in "few thousand days"
- Recently stated physicists will be "superfluous"
- Marc Andreessen (Techno-Optimist Manifesto) believes AI will develop complete physics understanding through data/processing cycles
- Dario Amodei (Anthropic CEO) - unlikely scaling laws will stop
- Leopold Aschenbrenner predicts superhuman AI by 2027
Evidence of Scaling Problems
- OpenAI Orion model leaked information:
- Currently training but not reliably better than predecessor
- Better at language tasks but may not outperform at coding
- Ilya Sutskever (former OpenAI) told Reuters: pre-training scaling results have plateaued
- Google Gemini iteration not meeting internal expectations (3 sources)
- Twitter/X rumors: frontier labs hit "unexpected huge wall of diminishing returns"
Technical Analysis of Limitations
- Altman's optimism based on remaining optimizations ("unhobbling")
- Model iteration and thinking improvements (like newest GPT)
- Only delays problem if models don't scale with more data
- Fundamental physics problem: cannot deduce underlying reality from emergent level
- Example: Aristotle couldn't deduce standard model from observation
- Required particle colliders for real data
- Decoupling of scales between reality details and descriptions
- Real-world data severely limited for AI training
Expert Opinions
- Gary Marcus: predicted this years ago, unsurprised by current developments
- John LeCun (Meta) and Bill Gates: also speculated about scaling limits
- Yann LeCun quote analysis: misconception that predicting next token means understanding underlying reality
- Author's view: training LLMs like gym - adding more weights eventually counterproductive
Data Availability Issues
- Insufficient real-world data for AI training
- AI people claim data not a problem vs author's disagreement
- Videos/images are extremely limited representations of real world
- Need real-world data to break current impasse
