AI summary by TwinMind

How Gwern saw AI scaling coming

Dwarkesh Patel

5m 44s

Early Skepticism of Connectionist Arguments (Mid-2000s)

  • Read Ray Kurzweil and Hans Moravec's fundamental connectionist argument
    • Claimed sufficient computing power would discover neural network architecture matching human brain
    • Until that computing power available, AI seemed futile
  • Found argument very unlikely - "build it and they will come" view of progress
    • Seemed ludicrous that big supercomputer matching human brain would summon correct algorithms
    • Algorithms are complex, hard, require deep insight
    • Can't just buy computers and expect advanced AI - "magical thinking"
  • Knew argument but was super skeptical, didn't pay much attention

Shane Legg's Predictions and Growing Interest

  • Shane Legg very big on connectionist view in following years
  • As part of interest in transhumanism, LessWrong, AI risk - paid close attention to Legg's blog posts
  • Legg extrapolating trends with updated numbers from Kurzweil and Moravec
  • Precise predictions:
    • First generalist system around 2019 as Moore's law continues
    • By 2025: humanish agents with generalist capabilities
    • By 2030: AGI

Deep Learning Breakthroughs Change Perspective

  • DanNet and AlexNet came out - very impressive success story for connectionism
  • Question: isolated success story or what Kurzweil/Moravec/Legg predicted?
  • Started thinking trend worth keeping eye on, maybe not as stupid as originally thought
  • Kept reading deep learning literature, noticed patterns:
    • Dataset sizes kept getting bigger
    • Models kept getting bigger
    • GPUs slowly ramped up from one cheap consumer GPU to two, then eight
    • Neural networks expanded from niche individual use cases to broader applications
  • Gradual realization: "wow is there anything CNNs can't do?"
  • Daily ArXiv papers showing CNN applications - gradual trickle kept hitting
  • Maybe intelligence really is just compute + data + parameters
  • Maybe Moravec, Legg, Kurzweil were right

GPT Series Confirms Scaling Hypothesis

  • GPT-1 release: unsupervised sentiment neuron learning on its own - pretty amazing
  • Very compute-centric view: build Transformer, intelligence will come
  • GPT-2 release: "holy shit moment" - prompting and summarization capabilities
  • "Holy shit, do we live in their world?"

GPT-3 as Crucial Test

  • GPT-3 was crucial test - huge scale-up, one of biggest in neural network history
  • Not narrow specific task like Go - seemed like crucial test for scaling
  • If scaling bogus: GPT-3 paper would be totally unimpressive
  • If scaling true: automatically guaranteed much more impressive results than GPT-2
  • Opened first/second page, saw few-shot learning chart
  • "Holy shit, we are living in the scaling world, Legg and Moravec and Kurzweil were right"

Public Reaction Disappointment

  • Turned to Twitter - everyone else saying scaling works badly, not state-of-the-art
  • Made researcher really angry - "someone was wrong on the internet"
  • Had to write all this stuff up

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome