AI summary by TwinMind

AI Scaling Hits Wall, Rumours Say. How Serious is it?

Sabine Hossenfelder

7m 26s

AI Industry Claims vs Reality

  • Sam Altman (OpenAI) claims AI will solve all physics problems
    • September essay "The Intelligence Age" - confident AI gets better with scale
    • Predicts superintelligence in "few thousand days"
    • Recently stated physicists will be "superfluous"
  • Marc Andreessen (Techno-Optimist Manifesto) believes AI will develop complete physics understanding through data/processing cycles
  • Dario Amodei (Anthropic CEO) - unlikely scaling laws will stop
  • Leopold Aschenbrenner predicts superhuman AI by 2027

Evidence of Scaling Problems

  • OpenAI Orion model leaked information:
    • Currently training but not reliably better than predecessor
    • Better at language tasks but may not outperform at coding
  • Ilya Sutskever (former OpenAI) told Reuters: pre-training scaling results have plateaued
  • Google Gemini iteration not meeting internal expectations (3 sources)
  • Twitter/X rumors: frontier labs hit "unexpected huge wall of diminishing returns"

Technical Analysis of Limitations

  • Altman's optimism based on remaining optimizations ("unhobbling")
    • Model iteration and thinking improvements (like newest GPT)
    • Only delays problem if models don't scale with more data
  • Fundamental physics problem: cannot deduce underlying reality from emergent level
    • Example: Aristotle couldn't deduce standard model from observation
    • Required particle colliders for real data
  • Decoupling of scales between reality details and descriptions
  • Real-world data severely limited for AI training

Expert Opinions

  • Gary Marcus: predicted this years ago, unsurprised by current developments
  • John LeCun (Meta) and Bill Gates: also speculated about scaling limits
  • Yann LeCun quote analysis: misconception that predicting next token means understanding underlying reality
  • Author's view: training LLMs like gym - adding more weights eventually counterproductive

Data Availability Issues

  • Insufficient real-world data for AI training
  • AI people claim data not a problem vs author's disagreement
  • Videos/images are extremely limited representations of real world
  • Need real-world data to break current impasse

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome