AI summary by TwinMind
How Gwern saw AI scaling coming
Dwarkesh Patel
5m 44s
Early Skepticism of Connectionist Arguments (Mid-2000s)
- Read Ray Kurzweil and Hans Moravec's fundamental connectionist argument
- Claimed sufficient computing power would discover neural network architecture matching human brain
- Until that computing power available, AI seemed futile
- Found argument very unlikely - "build it and they will come" view of progress
- Seemed ludicrous that big supercomputer matching human brain would summon correct algorithms
- Algorithms are complex, hard, require deep insight
- Can't just buy computers and expect advanced AI - "magical thinking"
- Knew argument but was super skeptical, didn't pay much attention
Shane Legg's Predictions and Growing Interest
- Shane Legg very big on connectionist view in following years
- As part of interest in transhumanism, LessWrong, AI risk - paid close attention to Legg's blog posts
- Legg extrapolating trends with updated numbers from Kurzweil and Moravec
- Precise predictions:
- First generalist system around 2019 as Moore's law continues
- By 2025: humanish agents with generalist capabilities
- By 2030: AGI
Deep Learning Breakthroughs Change Perspective
- DanNet and AlexNet came out - very impressive success story for connectionism
- Question: isolated success story or what Kurzweil/Moravec/Legg predicted?
- Started thinking trend worth keeping eye on, maybe not as stupid as originally thought
- Kept reading deep learning literature, noticed patterns:
- Dataset sizes kept getting bigger
- Models kept getting bigger
- GPUs slowly ramped up from one cheap consumer GPU to two, then eight
- Neural networks expanded from niche individual use cases to broader applications
- Gradual realization: "wow is there anything CNNs can't do?"
- Daily ArXiv papers showing CNN applications - gradual trickle kept hitting
- Maybe intelligence really is just compute + data + parameters
- Maybe Moravec, Legg, Kurzweil were right
GPT Series Confirms Scaling Hypothesis
- GPT-1 release: unsupervised sentiment neuron learning on its own - pretty amazing
- Very compute-centric view: build Transformer, intelligence will come
- GPT-2 release: "holy shit moment" - prompting and summarization capabilities
- "Holy shit, do we live in their world?"
GPT-3 as Crucial Test
- GPT-3 was crucial test - huge scale-up, one of biggest in neural network history
- Not narrow specific task like Go - seemed like crucial test for scaling
- If scaling bogus: GPT-3 paper would be totally unimpressive
- If scaling true: automatically guaranteed much more impressive results than GPT-2
- Opened first/second page, saw few-shot learning chart
- "Holy shit, we are living in the scaling world, Legg and Moravec and Kurzweil were right"
Public Reaction Disappointment
- Turned to Twitter - everyone else saying scaling works badly, not state-of-the-art
- Made researcher really angry - "someone was wrong on the internet"
- Had to write all this stuff up
