AI summary by TwinMind
Jay McClelland: Neural Networks and the Emergence of Cognition | Lex Fridman Podcast #222
Lex Fridman
2h 31m 57s
Introduction and Background
- Jay McClelland: cognitive scientist at Stanford, seminal figure in AI/neural networks history
- Co-wrote Parallel Distributed Processing book with David Rumelhart
- Rumelhart co-authored backpropagation paper with Jeff Hinton
- Their work foundational to neural network-based machine learning revolution of past 15 years
Beauty of Neural Networks
- Most beautiful aspect: linking biology with mysteries of thought
- Late 60s/early 70s: cognitive psychology emerging as field, book "Cognitive Psychology" (1967) claimed nervous system study only peripheral interest
- McClelland disagreed - felt mind emerges from physical being ("dust to dust, ashes to ashes")
- Concept of "awakening from the Cartesian dream"
Descartes and Mind-Body Problem
- Descartes' hydraulic statue story at Versailles gardens - stepping on stones moved statues via water pressure
- Developed mechanistic theory of animal behavior through hydraulic "fibers"
- Believed humans had animal body plus divine element for thought
- Physical world included body/action but not thought according to Descartes
- Study of physiology focused on sensory/motor systems, cognition tied to abstract algorithms
Evolution and Intelligence
- Darwin's struggle with complexity arguments (eye development)
- Had nightmares about complexity, delayed publication due to fear of own ideas
- 20th century: evolution accepted for producing range of animals
- Continuity of species (apes, chimpanzees, bonobos) challenged Descartes' view
- Huxley had to demonstrate hippocampus exists in chimpanzee brains (late 19th century)
Language and Human Intelligence
- Chomsky argued genetic fluke 100,000 years ago created language capability
- Language provided special separation from other animals
- McClelland: language comes with evolution of sociality, mutual engagement, rich world-organizing mechanisms
- Language as tool for collective intelligence
- Evolution: punctuated equilibrium - long stasis periods followed by sudden changes
- Similar pattern in child development (Piaget's stage theory)
Early Career and Neural Network Discovery
- 1974: Assistant professorship at UCSD
- Rumelhart had written "Explorations in Cognition" with Don Norman
- Created spirit of playful exploration of ideas
- James Anderson conference talk: used linear algebra to create neural network models of perception/categorization/memory
- 1977: Walking across campus revelation - "if I think about the mind in terms of a neural network it will help me answer questions about the mind"
Parallel Distributed Processing Concepts
- "Parallel": each neuron as independent computational unit
- Billions of little computational units working simultaneously vs single central processor
- Revolutionary compared to sequential computer science approaches
- Convolutional neural networks: layers of thousands/tens of thousands of neurons
- Each neuron contributes tiny bit to whole computation
David Rumelhart Background
- From small town western South Dakota
- Mother: librarian, Father: newspaper editor
- Four competitive brothers encouraged by father
- Standout undergraduate, went to Stanford for mathematical psychology PhD
- Mathematical psychology: using mathematics/equations to model mental processes
- One of first assistant professors at UCSD psychology department (1967)
Interactive Model of Reading
- Rumelhart's breakthrough paper on reading comprehension
- Every level of analysis depends on all other levels
- Pixels → letters → words → ideas (hierarchical but interactive)
- Top-down and bottom-up processing simultaneously
- Experts at each level updating hypotheses at other levels
- Completely interactive bi-directional parallel distributed process
Connectionism Definition
- Knowledge exists in connections between units, not separate dictionary
- Unit for word "time" defined by connections to constituent letters
- Knowledge distributed across connection weights
- Counter-intuitive - knowledge not explicitly stored anywhere
Semantic Dementia and Rumelhart's Decline
- Progressive neurological condition affecting meaning comprehension
- Starts with inability to differentiate similar categories (dog breeds)
- Progresses to calling all medium animals "dogs", small ones "cats", large ones "horses"
- Loss of conceptual grounding from all input modalities
- Pyramids and Palm Trees test: can't relate concepts (pyramids go with palm trees vs pine trees)
- Rumelhart suffered from this condition - tragic irony given his work on distributed representation
- Unlike Hawking (motor system affected), Rumelhart's semantics/self-semantics affected
- Could still bowl, navigate spatially, point to menu items but couldn't name them
PDP Research Group Formation (1982)
- Hinton returned to UCSD on sabbatical
- Formed PDP research group with McClelland and Rumelhart
- Francis Crick joined from Salk Institute
- Paul Smolensky and others also involved
Backpropagation Development
- Hinton's key insight: stop looking to biology for learning rules
- Instead: define problem, figure out how to adjust connection weights to solve it
- Objective function approach - measure error/loss, minimize through weight adjustment
- Gradient descent already known for single layer (Widrow-Hoff delta rule)
- Rumelhart generalized to multiple layers - "generalized delta rule"
- Later called "backpropagation" - error signals propagated backward through connections
- Hinton initially skeptical, preferred Boltzmann machine approach
- Backprop worked better than Boltzmann machine learning
Jeff Hinton's Contributions
- Two 1981 papers: essentially transformers concept, early semantic cognition work
- 1977 unpublished paper: neural networks doing recursive computation via fast connection weight changes
- Visual/intuitive thinking style - draws pictures instead of equations in lab meetings
- Uses geometric metaphors (hands showing ravine shape for gradient descent)
- Descendant of logician Boole, long line of English academics
- Wanted to contribute to reasoning itself, not just human reasoning
- Boltzmann machine: connecting Boolean tradition with continuous probabilistic constraint satisfaction
Computational Intelligence
- McClelland uses term "computational intelligence" to characterize field
- Human-oriented computational intelligence researcher
- Interested in human solution but also going beyond human nervous system limitations
- Deep learning scaling human intelligence mechanisms through computational power
Mathematical Cognition
- Mathematics as tools for exploring idealized worlds often relevant to real world
- Objects with idealized properties, precise relationships
- Natural numbers as cultural construction, not innate endowment
- Tristan Needham quote: studying math as symbol manipulation like studying music without hearing notes
- Mathematics: development of systems for thinking about properties/relations among idealized objects
- Mathematical notation just way of expressing propositions, not deep representation
Mathematical Reasoning and Intuition
- Poincaré quote: "by logic we prove, by intuition we discover"
- Combination of intuitive connectionist knowledge with formal reasoning
- Flash of insight from constellation of simultaneous constraints
- Archimedes, Newton, Einstein, Hinton examples of sudden insights
- Deep learning systems showing similar creative synthesis (text generation, novel story creation)
- Alpha Zero chess: novel brilliant moves combining intuition with search
Formal vs Natural Thinking
- 3000+ years of formal system development (Thales, Euclid's Elements)
- Academia created modes of thought that become deeply ingrained
- Chomsky example: born into academic family, deeply enculturated in formal reasoning
- Lila Gleitman's discovery: linguists' intuitions about language differ from ordinary people
- Formally trained academics have systematicity-organized engagement with experience
- Different from natural human mind without formal immersion
Expert Blind Spot
- Things self-evident to experts that they can't explain to non-experts
- Kronecker: "God made natural numbers, all else is work of man" - wrong, natural numbers are cultural construction
- Mathematical ability often attributed to biological differences rather than dinner table conversations
- Beginner's mind vs expert limitations
Introspection and Self-Knowledge
- Nisbet and Wilson paper: limits on awareness of factors causing our choices
- Explanations of behavior often post-hoc rationalizations, not actual causal processes
- Our insight into beliefs/attitudes/feelings subject to culturally transmitted understanding
- Ability to introspect itself product of culture and belief practice
Life Advice
- Find what you're intrinsically motivated to engage with
- Celebrate that discovery when it happens
- McClelland's wandering undergraduate years until 1968 Columbia revolution
- Psychology offered chance to run own experiments (vs predetermined physics/chemistry labs)
- Intrinsic motivation leads to immersion, which creates expertise opportunity
- "Mozart effect": synergy of innate resonance with intensive experience
- Different siblings found different passions despite shared environment
Mortality and Legacy
- More afraid of degeneration than death itself
- Fear of losing ability to engage at cutting edge of ideas
- Motivated by limited time to tackle hard problems (mathematical cognition)
- Legacy: contributing to science through collaborative opportunities
- Joy of science: partially formed thoughts crystallizing through discourse into scientific progress
- Interactive activation model and backpropagation as examples
- Don't let others label you - continue growing beyond assigned categories
Meaning of Life
- Humans make meaning for themselves
- Inspired by meaning others made in past (religious thinkers of first millennium BC)
- Emergent result of natural process without guidance
- Meaning is what we make of it, not what we find
- Very personal and individual - comes through particular combination of fabric, experience, context
- Local context creates peculiar emergent processes that tell stories about themselves
